←搜索结果

搜索“DeepSeek”

按匹配度 · 共 1368 条结果 清除搜索

星期一

  1. vLLM 官方博客

    vLLM 分布式推理中的张量并行与流水线并行

    vLLM 介绍了如何结合张量并行和流水线并行，在多 GPU 或多节点上部署超出单卡容量的大模型。张量并行可降低延迟但依赖高带宽互连，流水线并行通信开销较低且更适合跨节点部署。案例中从 TP=1 扩展到 TP=2 后，KV cache block 增加 13.9x，token 吞吐量提升 3.9x。

    idth interconnects between each GPU, like NVLink or InfiniBand, to minimize overhead from the increased communication costs. Pipeline Parallelism For extremely large models (e.g., DeepSeek R1, Llama 3.1 405B), a single node may not suffice. Pipeline parallelism shards models across nodes, each handling specific contiguous model layers. How It Works Each GPU loads and processes a distinct set of layers; Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation

    匹配字段正文

    #engineering#教程/实践#部署/工程#推理

星期四

  1. Mistral AI精选

    Mistral AI 发布 24B 参数模型 Mistral Small 3

    Mistral AI 发布延迟优化的 24B 参数模型 Mistral Small 3，并以 Apache 2.0 许可开放预训练和指令微调检查点。官方称其 MMLU 准确率超过 81%、推理速度达 150 tokens/s，与 Llama 3.3 70B instruct 表现相当且在同一硬件上快 3x 以上。量化后可在单张 RTX 4090 或配备 32GB RAM 的 Macbook 上本地运行，并已通过 la Plateforme 及合作平台提供。

    Up and to the Left

    a powerful base for accelerating progress. Note that Mistral Small 3 is neither trained with RL nor synthetic data, so is earlier in the model production pipeline than models like Deepseek R1 (a great and complementary piece of open-source technology!). It can serve as a great base model for building accrued reasoning capacities. We look forward to seeing how the open-source community adopts and customizes it. Performance Human Evaluations We conducted side by side evaluations with an external t

    匹配字段正文

    #lab#模型发布#部署/工程#开源生态

星期一

  1. vLLM 官方博客精选

    vLLM V1 Alpha 发布，重构核心架构并将吞吐量提升至多 1.7x

    vLLM 发布 V1 Alpha，通过重构调度器、KV cache 管理器、Worker、采样器和 API 服务器，将吞吐量较 V0 提升至多 1.7x。V1 默认启用低开销前缀缓存，增强多模态模型支持，并可通过设置 `VLLM_USE_V1=1` 在不修改现有 API 的情况下启用。当前版本仅支持 Ampere 或更新的 NVIDIA GPU，尚不支持 LoRA、结构化解码、推测解码和流水线并行等功能。

    implementing the speculative decoding support; Kuntai Du is implementing the prefill disaggregation and KV Cache transfer support; Simon Mo and Zhuohan Li contributed to the V1 system design Related Posts DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

    匹配字段正文

    #engineering#产品更新#部署/工程#多模态

星期二

  1. Prime Intellect精选

    Prime Intellect 发布 INTELLECT-MATH 数学推理模型与 500 万条推理轨迹数据集

    Prime Intellect 通过 QwQ-32B 生成并校验合成推理轨迹，微调 Qwen-7B-Math 后再用 PRIME-RL 做在线强化学习，训练出 7B 参数数学推理模型 INTELLECT-MATH，其早期 checkpoint 以 47 个 RL step 达到 Eurus-2-7B-PRIME 用 592 步的效果，训练时间缩短 10 倍。团队同时开源代码、权重和 NuminaMath-QwQ-CoT-5M 数据集，该数据集包含 86 万道数学题上的 500 万条经正确性校验的推理轨迹。文章还提出在推理时计算范式下，RL 中大量前向计算会降低通信需求，使全球分布式训练成为可行路径，并计划今年训练 o3 的完全开源复现。

    Distributed Training in the Inference-Time-Compute Paradigm

    policy initializations for online reinforcement learning. We use this insight to train INTELLECT-MATH, a state-of-the-art model for mathematical reasoning (prior to the release of Deepseek-R1) in the 7B parameter regime that reduces the training time required to match the existing state-of-the-art by 10x. Additionally, we release NuminaMath-QwQ-CoT-5M, a dataset containing five million reasoning traces across 860k mathematics questions verified for correctness. All of our code as well as weights

    匹配字段正文

    #company#模型发布#推理#数据/训练

星期二

  1. vLLM 官方博客

    vLLM 结构化解码入门与 XGrammar 集成解析

    vLLM 介绍结构化解码的原理与实现，并已支持 Outlines 和 XGrammar 两种后端，以约束大语言模型输出符合指定结构。XGrammar 集成在负载下可将每个输出 token 的生成时间改善至多 5x，不适用的请求会回退至 Outlines。规划中的 v1 将重点优化性能，并探索调度器级掩码广播以支持混合请求批处理。

    top level view of structure decoding

    o the same mechanism of "using a format for the model to structurally sampling outputs.” ↩; See this blog post from HuggingFace for using logit processors to control the generation process. ↩ Related Posts DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

    匹配字段正文

    #engineering#教程/实践#部署/工程#Agent

星期四

  1. vLLM 官方博客

    vLLM v0.6.0 发布，吞吐量提升最高 2.7x、TPOT 最多降低 5x

    vLLM 发布 v0.6.0，相比 v0.5.3 吞吐量提升 1.8-2.7x，其中 Llama 8B 的吞吐量提升 2.7x、TPOT 加快 5x，Llama 70B 的吞吐量提升 1.8x、TPOT 减少 2x。主要优化包括拆分 API server 与推理引擎、多步批量调度、异步输出处理、对象缓存和非阻塞数据传输。多步调度目前会造成 token 间延迟波动，并可能在低请求率下提高 TTFT。

    has highest throughput on ShareGPT and Decode-heavy dataset (though the throughput is only marginally higher than TensorRT-LLM), but the throughput of vLLM is lower on Prefill-heavy dataset. Related Posts DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

    匹配字段正文

    #engineering#产品更新#部署/工程#开源生态

星期二

  1. vLLM 官方博客

    vLLM 与 DeepSpeed-FastGen 性能对比分析

    vLLM 在常见场景中速度与 DeepSpeed-FastGen 相当，处理较长输出时更快，其他场景最高可快 1.8x。DeepSpeed-FastGen 仅在长提示词、短输出负载中凭 Dynamic SplitFuse 占优，但实测增益未达其宣称的 2x。

    ing popular features like stop strings and parallel sampling (e.g., beam search). We do expect the DeepSpeed-FastGen is eager to catch up and we welcome the creative innovation in the market! Related Posts DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

    匹配字段正文

    #engineering#评测/基准#部署/工程#开源生态

星期二

  1. vLLM 官方博客精选

    vLLM 发布基于 PagedAttention 的开源 LLM 推理与服务库

    vLLM 发布开源 LLM 推理与服务库，利用 PagedAttention 管理 KV cache，无需修改模型架构即可实现比 HuggingFace Transformers 最高 24x 的吞吐量。实验中其吞吐量比 TGI 最高提升 3.5x，内存浪费降至 4% 以下；LMSYS 部署后将服务所需 GPU 数量减少 50%。

    ire team — Siyuan Zhuang, Ying Sheng, Lianmin Zheng (UC Berkeley), Cody Yu (Independent Researcher), Joey Gonzalez (UC Berkeley), Hao Zhang (UC Berkeley & UCSD), and Ion Stoica (UC Berkeley). Related Posts DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.

    匹配字段正文

    #engineering#产品更新#部署/工程#开源生态