vLLM 与 DeepSpeed-FastGen 性能对比分析
Notes on vLLM v.s. DeepSpeed-FastGen | vLLM Blog
vLLM 在常见场景中速度与 DeepSpeed-FastGen 相当，处理较长输出时更快，其他场景最高可快 1.8x。DeepSpeed-FastGen 仅在长提示词、短输出负载中凭 Dynamic SplitFuse 占优，但实测增益未达其宣称的 2x。
正文
当前显示原文，中文全文译文尚未就绪。
TL;DR:
vLLM matches DeepSpeed-FastGen's speed in common scenarios and surpasses it when handling longer outputs; DeepSpeed-FastGen only outperforms vLLM in scenarios with long prompts and short outputs, due to its Dynamic SplitFuse optimization. This optimization is on vLLM’s roadmap; vLLM’s mission is to build the fastest and easiest-to-use open-source LLM inference and serving engine. It is Apache 2.0 and community-owned, offering extensive model and optimization support
The DeepSpeed team recently published a blog post claiming 2x throughput improvement over vLLM, achieved by leveraging the Dynamic SplitFuse technique. We are happy to see the technology advancements from the open-source community. In this blog, we show the specific scenarios where the Dynamic SplitFuse technique is advantageous, noting that these cases are relatively limited. For the majority of workloads, vLLM is faster than (or performs comparably to) DeepSpeed-FastGen.
Performance Benchmark
We've identified two key differences between vLLM and DeepSpeed-FastGen in terms of performance optimization:
DeepSpeed-FastGen adopts a conservative/suboptimal memory allocation scheme, which wastes memory when output lengths are large; DeepSpeed-FastGen’s Dynamic SplitFuse scheduling gives speedup only when prompt lengths are much greater than output lengths
As a result, DeepSpeed-FastGen outperforms when the workload is consistently long prompt and short output. In other scenarios, vLLM shows superior performance.
We benchmarked the two systems on an NVIDIA A100-80GB GPU with the LLaMA-7B model in the following scenarios:
Here, DeepSpeed-FastGen's Dynamic SplitFuse scheduling is expected to shine. However, the performance gain we observe isn't as significant as 2x.
Scenario 2: Other cases
In these cases, vLLM is up to 1.8x faster than DeepSpeed-FastGen.
We are committed to making vLLM the best open-source project incorporating the community’s best models, optimizations, and hardware. Coming out of UC Berkeley Sky Computing Lab, we are building vLLM truly in open source with the Apache 2.0 license.
The vLLM team prioritizes collaborations and we strive to keep the codebase with high quality code and easy to contribute. We are actively working on system performance; as well as new features like LoRA, Speculative Decoding, and better Quantization Support. Additionally, we are collaborating with hardware vendors like AMD, AWS Inferenetia, and Intel Habana to bring LLM to the broadest community.
Appendix: Feature Comparison
DeepSpeed-FastGen currently offers basic functionalities, supporting only three model types and lacking popular features like stop strings and parallel sampling (e.g., beam search). We do expect the DeepSpeed-FastGen is eager to catch up and we welcome the creative innovation in the market!
Related Posts
DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0
Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.