Google: Agentic AI Drives Up Data Center CPU Demand

Google: Agentic AI Drives Up Data Center CPU Demand Source: Semiconductor Horizons Published: 2026-11-02 11:00 Google CPU-generated poster There is no one-size-fits-all CPU that suits all types of agents.

Mark Lohmeyer, Vice President and General Manager of AI and Compute Infrastructure at Google Cloud, said in an interview that the various demands of agentic AI are imposing entirely new requirements on data center CPUs and even the entire infrastructure stack. "Conversational tasks are relatively simple: one question, one answer, with the compute load highly concentrated on GPUs or TPUs for inference. But the interaction model of agents has changed a great deal."

When using agents, a user's high-level instruction can be broken down into multiple subtasks and distributed to a group of agents. These agents can call other APIs, access web pages, and write and run code — all of which are CPU tasks. Conversation sessions can last much longer and can support multiple human users, meaning context and state may need to be retained for days or even weeks.

"Agent workloads tend to be bursty and sit idle most of the time," he said. "There may be no activity for a day or two until the user issues another instruction, or some API returns a result... It is different from traditional CPU workloads: state needs to be maintained for a long time, but it is often idle during that period; once it starts running, compute demand spikes sharply."

Lohmeyer said that how to carry such workloads while balancing performance and security is a major challenge facing the industry today. "The demand is already high and growing rapidly. What kind of infrastructure should we build to support this growth while keeping costs under control?"

Google's self-developed Arm-based Axion CPU

An entirely new workload profile

Lohmeyer distinguished two types of compute demand: one is the compute that runs the agents themselves, which is an entirely new workload requiring new supporting infrastructure; the other is tasks such as accessing web pages and calling APIs, which already existed but were previously triggered by humans. "Agent workloads will significantly affect both."

The scale of the existing portion of these tasks will rise sharply: agents make calls far more frequently than humans, and hundreds or thousands of agents can come online quickly, while the growth rate of human users cannot come close to that level.

Lohmeyer said: "For latency and performance reasons, ideally these agents, their corresponding context, applications and data should be as close to one another as possible."

However, he also noted that while proximity deployment is important, enterprise IT architectures will remain distributed, and at this stage most enterprises use multiple public clouds and private clouds at the same time. This means high-performance networking remains crucial, both within data centers and across data centers. "Agents with dynamic, stateful characteristics need a new type of computing infrastructure to support them," he said. "Agents need to access all kinds of existing applications and data, and this demand keeps rising; all components depend on network connectivity. Google is investing resources across all of these areas."

Lohmeyer said that for agentic AI, the optimal ratio of CPU to GPU compute will continue to adjust with workloads; but after shifting from foundation model training to agent inference, the share of CPU compute is clearly rising. "We expect both CPU and TPU/GPU workloads to continue growing rapidly. The CPU compute share has risen from a previously small portion to a level that cannot be ignored. As agents are deployed in more new scenarios and new environments, this ratio will continue to change," he said.

Co-design of hardware, software and workloads

Lohmeyer believes that solving the challenge of agent inference requires co-design across the entire infrastructure stack, covering CPU, storage, memory, networking and how all components work together. "Google Cloud has always adhered to the philosophy of optimizing infrastructure for workloads," he said. "In the agent era, this is more important than ever."

Because agent workloads are bursty and idle for long periods, agents can be "put to sleep": their running state is stored in storage media, freeing the CPU for other agents; when they need to resume, they are quickly loaded back. "This requires entirely new orchestration and scheduling mechanisms at the software level. In this architecture, it is crucial for servers to be equipped with high-performance storage to ensure fast wake-up," he said.

Google Cloud uses microVMs, equivalent to lightweight virtual machines, to set security isolation boundaries for multiple agents deployed on the same server. The same piece of hardware that could previously run only 100 traditional virtual machines can now host 1,000 microVMs.

"Security and control of agents are absolutely paramount, and Google is increasing investment at every layer of the technology stack," he said.

Lohmeyer said virtualization is the core foundation. Google plans to support multiple agent sandbox isolation approaches, among which microVMs are particularly critical. Compared with traditional virtual machines, microVMs streamline features, improve performance, and at the same time reduce the attack surface and enhance security. Google is also developing an agent gateway that can control agents' external access at the network layer, along with other higher-level control capabilities.

Lohmeyer mentioned that running thousands of microVMs simultaneously to host agents in the future will impose new requirements on core density, storage form factors, storage access methods and orchestration management software. The benefit of this approach is improved hardware utilization and lower operating cost per agent. "This can help customers expand their business, have a healthy business model and profitability, and avoid sacrificing performance while scaling," he said. "This type of capability will be very critical going forward."

CPU selection

Lohmeyer said there is no one-size-fits-all CPU that suits all types of agents. Google Cloud offers multiple x86 CPUs from Intel and AMD, as well as its self-developed Arm-based Axion CPU. "For data-intensive agent scenarios, you may need extremely strong single-core performance paired with large-capacity local SSDs; but for web-serving-oriented scenarios, customers will place more emphasis on core count," he said. "We believe the key in the end is to give customers choice, matching specific agent workloads to the most suitable CPU within Google Cloud."

Lohmeyer revealed that Axion is currently receiving a large amount of customer interest, including cloud instances optimized for performance and energy efficiency, as well as demand from large customers for bare-metal instances.

This article is reposted from media reports or online platforms and represents the author's personal stance or views. Our reposting is solely for sharing and does not mean we endorse or agree with it. If the source attribution is incorrect or your lawful rights and interests are infringed, please contact customer service promptly, and as a neutral platform service provider we will promptly correct, delete or handle it in accordance with the law. Comments No user comments yet Latest News More Related Recommendations

NVIDIA RTX 5080 24G introduced to China ahead of schedule, price may exceed 30,000 yuan 2026-11-02 11:42

A realistic cold reflection behind TSMC's 3D DRAM patent 2026-11-02 11:40

220 million yuan acquisition, Huada Empyrean completes a key link in digital EDA 2026-11-02 11:25