Arm proposes a CPU performance benchmarking methodology for agentic AI, with a white paper comparing AMD EPYC

On October 27, Chip Inside reported that recently, Ravi Malhotra, Director of Cloud AI Solutions Architecture at Arm, shared with Chip Inside and other media a new set of CPU performance evaluation methodologies that Arm has developed for agentic AI development, and summarized this methodology in the newly released whitepaper "How to Define CPU and Software Stack Performance for Agentic AI."

Before the emergence of agentic AI, the AI field mainly measured performance by Token generation, namely tokens per second, tokens per watt, and tokens per dollar. Agentic AI has changed this, requiring that the generated Tokens can be effectively applied in the real world. This means evaluating performance not only from the perspective of Tokens, but also from the perspective of the agent and the tasks it executes.

Overall, agentic tasks can be divided into two stages:

First, Token generation, namely reasoning and planning, which usually runs on AI racks, with the focus on large-scale Token generation and providing sufficient memory and storage space for the context and data interpretation required by the agent.

Second, taking actions based on Tokens, namely tool calling and orchestration, which usually runs on pure CPU servers, and the relevant servers and racks still need to meet the infrastructure requirements for large-scale deployment and stable operation.

The key question is, what kind of CPU can better carry such workloads?

In the whitepaper "How to Define CPU and Software Stack Performance for Agentic AI," Arm proposes a different approach to agentic AI performance evaluation. Its methodology mainly consists of two parts: component-oriented benchmarking and end-to-end agentic benchmarking.

Testing the overhead of a Web server, database, or model service can reflect the performance of a single component, but cannot reflect how these components are connected and operate collaboratively in an agentic workflow. This is where agentic benchmarking comes into play, providing a representative reference for evaluating end-to-end agentic workflows.

These component benchmarks and agentic benchmarks test and evaluate the CPU from different aspects. For example, tool-calling components focus on IPC (instructions per cycle), I/O, and concurrency; orchestration emphasizes CPU concurrency and utilization; reasoning and planning focus on testing whether the CPU has the ability to improve single-thread performance, including branch prediction, cache, memory latency, and memory bandwidth.

End-to-end agentic benchmarking focuses more on scalability and concurrency. Performance evaluation cannot only look at how many agents a single CPU can carry, nor can it simply pursue the highest throughput; it also needs to strike a balance between throughput and SLA (Service Level Agreement) requirements. This is what Arm emphasizes as sustained performance.

For software development workloads, running one to two agents per core may already be sufficient, and further increasing the number of concurrent agents may fail to meet SLA requirements. Enterprise operations or personal assistant workloads can support more concurrent agents, and the key still lies in how to balance throughput and SLA, which requires taking into account both single-core performance and the scalability of the entire CPU.

Arm focuses not only on performance at the CPU level, but also on the performance delivered at the rack level. Ravi Malhotra said that Arm's CPU design principle is to improve performance without increasing power consumption as a cost, and to pay more attention to balancing core-level performance, server-level scalability, and rack-level efficiency, whereas x86 architectures or other CPUs usually tend to emphasize one of these parameters.

An independent report released by Signal65, an independent technology analysis firm cooperating with Arm, compared the performance of Arm AGI CPU and the fifth-generation AMD EPYC processor (codenamed "Turin") on agentic workloads, focusing on the efficiency, scalability, and performance balance brought by Arm AGI CPU.

The report shows that for the evaluated agentic workloads, under the same power budget, the per-rack performance of Arm AGI CPU is about 1.99 times that of AMD EPYC 9755; from the perspective of tool calling, it reaches 1.7 times that of the comparison platform; on reasoning and planning applications such as vector databases and caching, it reaches 2.2 times that of the comparison platform; and from the end-to-end agentic benchmark results, it is 2 times that of the comparison platform.

For tool calling, Arm chose nginx for web services and MySQL for databases; for reasoning and planning, it chose the vector database Milvus and the model serving framework vLLM, and also chose fastText as the data processing benchmark; for orchestration, it measured sandbox execution overhead, selecting various sandbox technologies such as Docker, Firecracker, and gVisor to evaluate the additional performance cost of running each agent in a sandbox environment.

For agentic benchmarks, Arm chose one of the most popular application categories—software development—and selected SWE-bench and Terminal-Bench as representative references for software development workflows. On average, Arm AGI CPU is about 20% higher in performance at both the per-core and CPU SoC levels than the comparison platform.

One of the key advantages of Arm AGI CPU lies not only in performance, but also in efficiency and scalability. Because each core consumes less power, it can deploy more cores under the same rack power budget. Combining per-core and per-CPU performance with the larger number of cores per rack brings about a roughly 2x rack-level performance advantage.

Therefore, Arm benchmarked the time required to create these sandbox containers, testing different scenarios, including cold creation and warm creation, and also testing scenarios where multiple containers run in parallel on the same physical core, thereby forming oversubscription. The lower the sandbox overhead, the more customer support tickets the system can handle in the same amount of time, thereby improving overall efficiency and reducing the cost per task. In different scenarios, the overhead cost on Arm AGI CPU is far lower than that on AMD EPYC processors.

Arm is bringing these capabilities of Arm AGI CPU to market in different product forms, with two server reference designs—the Arm AGI CPU 1OU dual-node server reference design and the Arm AGI CPU 2U2P server reference design. Its partners include Supermicro, Lenovo, and ASRock, providing a range of production platforms and commercial servers in different sizes and configurations. In such use cases, Arm AGI CPU serves as the head node and works in coordination with the accelerators selected by customers. At the rack level, there are also different air-cooled and liquid-cooled solutions.

Conclusion: Agentic CPU performance evaluation depends on use cases

One of the biggest changes that agentic AI has brought to the level of CPU performance evaluation is greater attention to how components are combined in an efficient way to enable agentic AI operation. As use cases continue to evolve, choosing a different CPU for each use case is not a scalable model.

Arm recommends not focusing only on a single score, but comprehensively examining the performance of each component and evaluating the CPU platform according to one's own use cases—first determining the type of use case, and then selecting the corresponding end-to-end agentic benchmarks and components accordingly.