Zyphra 发布开源小模型 Zamba2-7B
Zyphra 发布 7B 规模小语言模型 Zamba2-7B,采用 Mamba2 与两个共享注意力块交替的混合架构,并以 Apache 2.0 开源权重。官方称其在评测基准上超过 Mistral-7B、Gemma-7B 和 Llama3-8B,首 token 时间快 25%、每秒 token 数提升 20%,内存占用也明显降低。模型基于 3 万亿 token 预训练数据,并在 128 张 H100 上训练约 50 天。

Zyphra is excited to release Zamba2-7B, a state-of-the-art small language model. At the 7B scale, we outperform the leading models of Mistral, Google’s Gemma and Meta’s Llama3 series in both quality and performance. Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B; Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in t
匹配字段正文
