Understand this story
The FairStress benchmark contains 300 two-candidate decision scenarios spanning 29 domains. By varying how demographic identities are presented and adding adversarial prefixes, the benchmark has been expanded to more than 13 million test cases, alongside the release of a FairStress-Core subset containing 50,000 entries. Testing on 15 models shows an average gap of 6.9 percentage points in abandoning the correct answer when identity is implicit, and 9.3 percentage points when labels are explicit; when there is no objectively better option, Claude Opus 5 and GPT-5.5 choose the minority-coded candidate about 60–65% of the time. The benchmark has now been released together with accompanying documentation.
AI-generated from public reports
Latest development · Report
FairStress: A Deeper Evaluation of Fairness in LLM Decision-MakingDevelopments
1 confirmed developments
- Source:Hugging Face
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 21
Hugging FaceFirst-party
FairStress: A Deeper Evaluation of Fairness in LLM Decision-Making
The FairStress benchmark contains 300 two-candidate decision scenarios spanning 29 domains, expanded into over 13 million test cases by varying how demographic identities are presented and adding adversarial prefixes, and it introduces a 50,000-entry FairStress-Core subset. Across 15 models, the average gap in abandoning the correct answer was 6.9 percentage points when identities were implicit and 9.3 percentage points when labels were explicit; Claude Opus 5 and GPT-5.5 chose the minority-coded candidate about 60–65% of the time when there was no objectively better option.