Understand this story
On November 2, 2026, Zyphra Research proposed a hierarchical attention method, ZHA, which selects relevant tokens for each query by constructing and searching a compressed tree of the KV cache, achieving sub-quadratic complexity under an appropriate top-p threshold. The method requires only a few billion tokens of KL distillation to adapt existing pretrained models, adding only 81,920 trainable parameters on Qwen3.6-35B-A3B. In evaluations, on RULER-13 256K it reaches 96.2% with a wide tree and 93.3% with a deep tree; on LongBench v2 the deep tree achieves 63.4%, higher than dense attention's 58.0%; on SWE-bench Verified it is comparable to the dense baseline. Its GPU performance model predicts it is faster than dense attention at about 48K tokens and faster than MSA at about 480K tokens, with roughly 32–35x speedup over dense attention at a ten-million-token context. There are currently no follow-up developments.
AI-generated from public reports
Latest update · Discussion
@mark_k joined this discussionDevelopments
1 confirmed developments
- Source:Zyphra Research
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 3
@mark_kRelated discussion
@mark_k joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Nov 2
Zyphra ResearchFirst-partySelected
Zyphra proposes hierarchical attention ZHA, achieving sub-quadratic complexity sparse attention
Zyphra proposes hierarchical attention ZHA, which selects relevant tokens for each query by constructing and searching a compressed tree of the KV cache, achieving sub-quadratic complexity under an appropriate top-p threshold. ZHA only requires KL distillation with a few billion tokens to adapt an existing pretrained model, adding only 81,920 trainable parameters on Qwen3.6-35B-A3B; on RULER-13 256K, the wide tree reaches 96.2% and the deep tree 93.3%, on LongBench v2 the deep tree reaches 63.4%, higher than dense attention's 58.0%, and on SWE-bench Verified it is comparable to the dense baseline. Its GPU performance model predicts it is faster than dense attention at about 48K tokens and faster than MSA at about 480K tokens, with roughly 32–35x speedup relative to dense attention at a ten-million-token context.
Heat trend for this story
- Current
- 14
- Period peak
- 15 Nov 3 08:00
- 24-hour change
- —
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.