Inferact 谈 vLLM 跑智能体生产负载
Agent traffic is a strange workload for an inference server.
Inferact 的 Yifan Qiao 在 DigitalOcean Open Intelligence Summit 的 Serve 分场分享如何在智能体生产负载上运行 vLLM。智能体每轮都会重发几乎相同的长前缀（系统提示词、工具 schema、历史），只要求返回一个短工具调用，因此前缀缓存和 KV-cache 复用比原始 tokens per second 更关键；长前缀缓存未命中意味着长任务每一轮都要重新支付完整 prefill 成本。
查看原帖Who gets to shape the future of AI? Today we're bringing together some of the biggest voices in open models, data, and agents to answer that question, including @rbren_dev of @OpenHandsDev, @zlxi02 of @inferact, @GerryFPatia of @boltdotnew, @vinaygo of @Amplitude_HQ and @latkins of @arcee_ai. The Open Intelligence Summit, co-hosted with @nvidia. Watch…
展开完整原文
Who gets to shape the future of AI? Today we're bringing together some of the biggest voices in open models, data, and agents to answer that question, including @rbren_dev of @OpenHandsDev, @zlxi02 of @inferact, @GerryFPatia of @boltdotnew, @vinaygo of @Amplitude_HQ and @latkins of @arcee_ai. The Open Intelligence Summit, co-hosted with @nvidia. Watch live, starting at 1 PM PT: Keynote: https://t.co/ioedT64hW3 Build: https://t.co/UcetmK4rtE Serve: https://t.co/vwoyYJQcn2 Run: https://t.co/fezeICypis
正文
当前显示原文，中文全文译文尚未就绪。
Agent traffic is a strange workload for an inference server.
Each turn resends a long, nearly identical prefix (system prompt, tool schemas, history) and asks for a short tool call back.
At DigitalOcean’s Open Intelligence Summit, Yifan Qiao of Inferact is giving a Serve track session on running vLLM on agentic production workloads.
That shape of traffic is why prefix caching and KV-cache reuse matter more for agents than raw tokens per second. A cache miss on a long prefix means paying for the full prefill again, on every turn of a long task.