AI Trends

Trending story

Zyphra Extends ZHA to Trillion-Parameter-Scale Open-Weight Models

1 reports1 story updatesUpdated Nov 17 08:00

Understand this story

AI overview

On November 17, 2026, Zyphra Research announced that it has extended its hierarchical attention method ZHA to open-weight models such as Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B, and MiMo-V2.6-Pro. According to the company, the conversion only requires freezing the backbone and training a small number of reference queries, costing less than running the full evaluation suite; after conversion, most evaluations are close to the original dense attention baseline, with Qwen dropping by less than 2 points on the 1M RULER. Zyphra also noted that the computational gains of ZHA depend on the model's own attention distribution: Qwen exhibits a clean sublinear power law, Nemotron and MiMo are nearly linear in the earliest few layers, and GPT-OSS shows an anomalously flat profile at the long-context end, with RULER 128K dropping from 63.3 to 46.0. There are currently no further developments.

AI-generated from public reports

Latest development · Report

Zyphra Scales ZHA Sparse Attention to Trillion-Parameter Models

Developments

1 confirmed developments

  1. Zyphra scales ZHA to trillion-parameter open-weight models

    Source:Zyphra Research

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 17

  1. Zyphra ResearchFirst-party

    Zyphra Scales ZHA Sparse Attention to Trillion-Parameter Models

    Zyphra applies its hierarchical attention method ZHA to open-weight models such as Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B, and MiMo-V2.6-Pro. After conversion, most evaluations are close to the original dense attention baseline, with Qwen dropping less than 2 points on 1M RULER. Conversion only requires freezing the backbone and training a small number of reference queries, costing less than running the full evaluation. However, ZHA's computational gains depend on the model's own attention distribution: Qwen exhibits a clean sublinear power law, Nemotron and MiMo are nearly linear in the earliest few layers, and GPT-OSS shows an abnormally flat profile at the long-context end, with RULER 128K dropping from 63.3 to 46.0.

Related stories

  1. Related storiesBase Labs studies whether gradient-aligned rewards can achieve online policy supervision
  2. Related storiesThinking Machines Lab introduces the Inkling 2 Small end-to-end 4-bit training recipe
  3. Related storiesAi2, in collaboration with multiple institutions, releases the XYBench benchmark test
  4. Related storiesZyphra proposes sparse attention method ZHA