Understand this story
On November 12, 2026, Apple Machine Learning Research published a study at the "Can We Trust AI Evaluation?" Workshop at NeurIPS 2026, evaluating the reliability of LLM judges on personalized writing style tasks. The study used four LLMs to generate texts for three types of long-form writing tasks, then collected human pairwise preference annotations and pairwise ranking annotations from judge LLMs and compared them. The results show that judging quality is only weakly correlated with general capability as measured by MMLU (Pearson's ρ ≤ 0.2), and varies with the writing task and evaluation criterion (general helpfulness vs. stylistic personalization); LLM judges using pairwise comparison are more consistent and perform better than independent scoring, and the "strongest" LLM was not the most reliable at reproducing human evaluations.
AI-generated from public reports
Latest development · Report
Apple Research: How LLM Judges Perform in Personalized Writing Style EvaluationDevelopments
1 confirmed developments
Apple research team publishes study on LLM judges for personalized writing evaluation
Source:Apple Machine Learning Research
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 12
Apple Machine Learning ResearchFirst-party
Apple Research: How LLM Judges Perform in Personalized Writing Style Evaluation
Apple published research at the NeurIPS 2026 "Can We Trust AI Evaluation?" Workshop, evaluating the reliability of LLM judges on personalized writing style tasks: using four LLMs to generate text for three types of long-form writing tasks, then collecting human pairwise preference annotations and judge LLM pairwise ranking annotations and comparing them. Results show that judging quality is only weakly correlated with general capability as measured by MMLU (Pearson's ρ ≤ 0.2), and varies with the writing task and evaluation criterion (general helpfulness vs. stylistic personalization). LLM judges using pairwise comparisons are more consistent and perform better than independent scoring, and the "strongest" LLM did not most reliably reproduce human evaluations.