腾讯 Hy4 preview 在 vLLM 上运行

@TencentHunyuan's Hy4-preview runs in vLLM from day 0, verified on NVIDIA GPUs. 🎉 - 770B total, 49B active, 256 routed experts plus one shared - 1M context, but each query attends to just 2048 tokens - Only 21 of the 78 layers compute their own sparse index, the other 57 reuse one - A 10B MTP layer ships inside the checkpoint, 0.7B of it active, draft…

展开完整原文

@TencentHunyuan's Hy4-preview runs in vLLM from day 0, verified on NVIDIA GPUs. 🎉 - 770B total, 49B active, 256 routed experts plus one shared - 1M context, but each query attends to just 2048 tokens - Only 21 of the 78 layers compute their own sparse index, the other 57 reuse one - A 10B MTP layer ships inside the checkpoint, 0.7B of it active, draft depth 3 Tencent's HPC-Ops attention and MoE kernels have been in vLLM main since Hy3. VLLM_ENABLE_HPC_OPS=1 vllm serve tencent/Hy4-preview-FP8 -tp 8 Thanks @TencentHunyuan for the preview weights! 🙌 🔗 https://t.co/REAxUUfyZb