AI News

Trending story

Apple research team publishes study on personalized writing evaluation by LLM reviewers

1 reports1 story updatesUpdated Nov 12 03:05

Understand this story

AI overview

On November 12, 2026, Apple Machine Learning Research published a study at the "Can We Trust AI Evaluation?" Workshop at NeurIPS 2026, evaluating the reliability of LLM judges on personalized writing style tasks. The study used four LLMs to generate texts for three types of long-form writing tasks, then collected human pairwise preference annotations and pairwise ranking annotations from judge LLMs and compared them. The results show that judging quality is only weakly correlated with general capability as measured by MMLU (Pearson's ρ ≤ 0.2), and varies with the writing task and evaluation criterion (general helpfulness vs. stylistic personalization); LLM judges using pairwise comparison are more consistent and perform better than independent scoring, and the "strongest" LLM was not the most reliable at reproducing human evaluations.

AI-generated from public reports

Latest development · Report

Apple Research: How LLM Judges Perform in Personalized Writing Style Evaluation

Developments

1 confirmed developments

  1. Apple research team publishes study on LLM judges for personalized writing evaluation

    Source:Apple Machine Learning Research

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 12

  1. Apple Machine Learning ResearchFirst-party

    Apple Research: How LLM Judges Perform in Personalized Writing Style Evaluation

    Apple published research at the NeurIPS 2026 "Can We Trust AI Evaluation?" Workshop, evaluating the reliability of LLM judges on personalized writing style tasks: using four LLMs to generate text for three types of long-form writing tasks, then collecting human pairwise preference annotations and judge LLM pairwise ranking annotations and comparing them. Results show that judging quality is only weakly correlated with general capability as measured by MMLU (Pearson's ρ ≤ 0.2), and varies with the writing task and evaluation criterion (general helpfulness vs. stylistic personalization). LLM judges using pairwise comparisons are more consistent and perform better than independent scoring, and the "strongest" LLM did not most reliably reproduce human evaluations.

Related stories

  1. Related storiesAnthropic launches Claude Fable 5.2 and Claude Mythos 5.2
  2. Related storiesAi2, in collaboration with multiple institutions, releases the XYBench benchmark test
  3. Related storiesLG AI Research proposes the dynamic evaluation protocol ATAD
  4. Related storiesAi2 releases novelty assessment research and benchmark