IBM 等维护方发布 llm-d 1.0,为自建模型推理增加运维控制
IBM、Red Hat 与 Google 等 llm-d 项目贡献者发布分布式 AI 推理框架 1.0 版本,新增多租户服务质量、零停机升级和面向 AI 智能体的服务模式。框架基于 Kubernetes,可与 vLLM、SGLang 等推理引擎配合,通过路由把请求导向已缓存相关上下文的副本,并在 GPU 内存、CPU 内存和存储之间迁移缓存。IBM 称在一项智能体编码测试中路由器捕获了 97% 的可用缓存复用,IBM 与 Red Hat 还在自建基础设施上并发运行 3000 个软件工程智能体,成本比通过托管服务运行同一开源模型低 5 到 10 倍。
o-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS. The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform. “At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in c
匹配字段正文

