Understand this story
On November 17, 2026, Zyphra Research announced that it has extended its hierarchical attention method ZHA to open-weight models such as Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B, and MiMo-V2.6-Pro. According to the company, the conversion only requires freezing the backbone and training a small number of reference queries, costing less than running the full evaluation suite; after conversion, most evaluations are close to the original dense attention baseline, with Qwen dropping by less than 2 points on the 1M RULER. Zyphra also noted that the computational gains of ZHA depend on the model's own attention distribution: Qwen exhibits a clean sublinear power law, Nemotron and MiMo are nearly linear in the earliest few layers, and GPT-OSS shows an anomalously flat profile at the long-context end, with RULER 128K dropping from 63.3 to 46.0. There are currently no further developments.
AI-generated from public reports
Latest development · Report
Zyphra Scales ZHA Sparse Attention to Trillion-Parameter ModelsDevelopments
1 confirmed developments
- Source:Zyphra Research
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 17
Zyphra ResearchFirst-party
Zyphra Scales ZHA Sparse Attention to Trillion-Parameter Models
Zyphra applies its hierarchical attention method ZHA to open-weight models such as Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B, and MiMo-V2.6-Pro. After conversion, most evaluations are close to the original dense attention baseline, with Qwen dropping less than 2 points on 1M RULER. Conversion only requires freezing the backbone and training a small number of reference queries, costing less than running the full evaluation. However, ZHA's computational gains depend on the model's own attention distribution: Qwen exhibits a clean sublinear power law, Nemotron and MiMo are nearly linear in the earliest few layers, and GPT-OSS shows an abnormally flat profile at the long-context end, with RULER 128K dropping from 63.3 to 46.0.
AI Trends