Understand this story
Google has published a new paper proposing the PLE benchmark for entity engineering tasks. The benchmark contains 85 electrical, civil, structural, and mechanical engineering tasks written or verified by practicing engineers. According to a report by Rohan Paul, the strongest agent scores only 50.6% on PLE: Claude Opus 5.5 in Claude Code ranks first among 14 models at 50.6%, consuming 27M tokens and 63 minutes per task, requiring reading about 61 project files and delivering schedules, reports, or 2D/3D drawings. The report also states that in design tasks its output satisfies 91% of constraints, but only 41.3% satisfy all hard constraints. There are currently no follow-up developments.
AI-generated from public reports
Developments
1 confirmed developments
- Source:Rohan Paul
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 29
Rohan Paul
Google's new paper proposes the PLE benchmark: the strongest agent scores only 50.6% on physical engineering tasks
Google's new paper proposes the PLE benchmark, comprising 85 electrical, civil, structural, and mechanical engineering tasks written or verified by practicing engineers; the strongest agent scores only 50.6%. Claude Opus 5.5, in Claude Code, ranks first among 14 models at 50.6%, consuming 27M tokens and 63 minutes per task, needing to read about 61 project files and deliver schedules, reports, or 2D/3D drawings. On design tasks, its output satisfies 91% of constraints, but only 41.3% satisfy all hard constraints.