AI 热点

Arena.ai 推出 Agent Arena Index 并更新智能体排行榜

Arena更新于 11月19日 03:11

The Agent Arena Index

AI 导读

Arena.ai 更新智能体排行榜,推出覆盖超过 2.9M 次用户会话的 Agent Arena Index,以及 Agentic Work、Agentic Coding、Agentic Chat、Task Outcome、Steering 和 Tool Use 等专项索引。新版分数缩放方式不会改变原有模型排名;Tool Hallucination 信号被移除,Confirmed Success 更名为 Task Completion,Alignment Index 保持不变。

推荐理由

AI 评分 54

新版索引拆分任务领域与智能体能力,并调整分数缩放方式,便于跨时间比较模型表现。

主题

标签

正文

We’re refreshing our Agent leaderboard to provide clearer scores and more useful benchmarks of agentic performance.

The headline leaderboard — the Agent Arena Index — provides a holistic view of agentic performance across all types of agentic work, spanning over 2.9M user sessions (and counting). We’re introducing several additional indexes focused on specific task domains and agentic skills: Agentic Work, Agentic Coding, Agentic Chat, Task Outcome, Steering, and Tool Use. The Alignment Index remains unchanged as a parallel measure alongside the Agent Arena Index.

In this article, we explain how our methodology has evolved since the previous Agent Arena Leaderboard and describe the new indexes we’re launching today.

Methodological updates

The methodology described in our original Agent Arena blog post remains the same at its core. The main difference is a new method for rescaling the scores that enables easier interpretation and comparison of model scores over time. This change does not affect the ranking. We’re also introducing a change to our signal set. We elaborate on both these changes below.

Signal changes

We are retiring the Tool Hallucination signal, as frontier models now rarely make the mistake of calling non-existing tools. All other signal definitions are unchanged. We are renaming the Confirmed Success signal to Task Completion for clarity, with no change to how it’s measured.

New score scaling

We are introducing a new score scaling that enables more interpretable comparisons of scores over time. The new scoring does not alter model rankings compared to the previous Agent Arena methodology. For a fixed model mmm, the previous methodology estimated its net improvement score:

Indexes

Below we describe which signals and domains enter each headline index and the new domain- and capability-specific indexes. Signals are combined into a single index through averaging, as before.

Agent Arena Index

The Agent Arena Index is a topline measure of agentic performance across all task categories and performance signals. It uses the Task Completion, Praise vs Complaint, Steerability, and Bash Reliability signals to give a holistic view of how well an agentic model gets real work done.

Alignment Index

The Alignment Index measures how well an agent aligns with the user's intentions and goals. It uses the Unauthorized Action, False Attribution, and Deceptive Completion signals across all task categories.

Domain and Capability Indexes

Each of these new indexes provides a focused view of agentic performance for a specific type of task or agent behavior. See breakdown of all indexes at our full blog: https://arena.ai/blog/introducing-the-agent-arena-index

来源

Arena