IBM 等维护方发布 llm-d 1.0，为自建模型推理增加运维控制

UPDATED 08:00 EDT / OCTOBER 20 2026

IBM-backed llm-d framework adds controls for running AI models in-house

by Paul Gillin

IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure.

The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.

Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are its lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.

The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform.

“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, wrote in e-mailed comments. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”

The project is focused on what happens after deployment. A production inference service must handle changing traffic, hardware failures, software updates and users competing for resources while maintaining acceptable response times.

Version 1.0 adds support for quality of service across multiple tenants, zero-downtime operations and serving patterns tailored to AI agents. Flow controls and prioritization are designed to keep bursts of agent activity from overwhelming services that require fast responses, such as coding assistants used by people.

The release “adds the maturity to run it like any other critical service: a stable [application programming interface], multiple tenants with competing priorities sharing the same infrastructure and upgrades that don’t interrupt operations,” said Carlos Costa, distinguished engineer at IBM Research and a project co-founder and lead maintainer.

Workloads can behave differently even when they reach models through the same endpoint. An agent may issue many requests, pause to use an external tool and return for additional model calls.

The framework can keep requests directed to appropriate infrastructure across multiple turns, increasing opportunities to reuse previously processed context. That matters because agents repeatedly consult the same code, documents, instructions and conversation history while completing assignments.

A routing system directs requests toward replicas that already hold relevant cached context. Another capability moves cached information among graphics processing unit memory, CPU memory and storage, preserving reusable information when GPU capacity becomes constrained.

These techniques were previously available, but version 1.0 integrates them with operational controls, along with methods that separate prompt processing from response generation, optimize work batches and distribute large mixture-of-experts models across accelerators. IBM cited a recent agentic coding test in which the framework’s router captured 97% of available cache reuse.

In a separate demonstration, IBM and Red Hat said they ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to 10 times lower than serving the same open model through a hosted service.

The economics become more significant as agents perform multiple rounds of reasoning, tool use and revision for a single assignment, IBM said. Reprocessing identical material at each step consumes additional computing resources.

Llm-d works with inference engines including vLLM and SGLang, supplying distributed routing and operational capabilities around model servers. It is currently hosted as a sandbox project by the Cloud Native Computing Foundation.

“It delivers high-performance, distributed inference with the reliability and scale enterprises need within an operational model they already know,” said Stephen Watt, vice president in Red Hat’s chief technology officer’s office.

Image: Pixabay

A message from John Furrier, co-founder of SiliconANGLE:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more

11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network

Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/

About SiliconANGLE Media

SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.

IBM 等维护方发布 llm-d 1.0，为自建模型推理增加运维控制

推荐理由

主题

标签

正文