RockAI Technology, together with HKUST and CMU, open-sources EdgeRock-4B, reducing KV Cache by 99.1%

A Google paper this year actually managed to stir up the stock prices of memory chip giants.

The paper mentioned that KV Cache can be compressed by at least 6x.

You know, large model inference is consuming more and more memory, and KV Cache is a considerable expense within that.

Now Google suddenly compressed it by this much, and the stock prices of memory chip giants like Micron fell accordingly.

But just a few months later, someone dealt an even harsher blow to KV Cache.

This time, compression alone wasn't enough.

As context grew from 4K all the way to 256K, KV Cache usage didn't change at all.

Under 256K context, it stays fixed at 72MiB. Compared to the source model, KV Cache is directly reduced by 99.1%.

Even more striking, a single 80GB GPU runs 512 concurrent long-reasoning streams, with output throughput reaching 12124 tokens/s, 6.42x that of the source model.

And reasoning ability is largely preserved: on AIME 2024–2026, the average pass@1 reaches 82.9%, while among the bounded-state models evaluated, even the best 14B model only reaches 18.8%.

This is EdgeRock-4B, just released and open-sourced by Yanxin Matrix under Yanshan Technology, together with HKUST, CMU and other universities.

One change it brings to long reasoning is:

No matter how long the reasoning, memory doesn't grow.

Both the model weights and inference code are now open-sourced, and it supports text and image reasoning.

Today's models casually think for tens of thousands of tokens; Agents can even work continuously for dozens of rounds, with search results, code, and tool-call records constantly being stuffed into context.

The models work harder and harder, but VRAM is the first to give out.

The problem lies in KV Cache.

Most models today store reasoning history in KV Cache, adding more with each generated token.

Take Qwen3.5-4B as an example: each token adds 32 KiB of KV Cache. At 64K reasoning, one sequence is 2 GiB; at 256K reasoning, it's 8 GiB.

This is especially critical in the era of reasoning models.

Models increasingly love "deep thinking," outputting tens of thousands of tokens at a time; an Agent runs dozens of rounds of tool calls, and context keeps rolling longer.

The longer the reasoning, the more VRAM is used, and the fewer requests a single card can serve simultaneously.

On the edge side, KV Cache also has to compete with perception and control modules for already limited memory.

The industry's approaches fall roughly into two categories.

One is to "slim down" each token:

Multi-head Latent Attention (MLA) compresses KV, quantization methods like KIVI reduce precision, and Google's TurboQuant belongs to this line too.

But they only reduce the cost per token; the cache still grows with reasoning.

The other is bounded-state models:

Recurrent models, state space models, sliding window models—memory is inherently fixed.

The problem is that they often can't retain distant details needed for subsequent reasoning, and reasoning ability suffers accordingly.

The test setup is 32K input, 32K output, with all models running on a single Hopper 80GB GPU, each using its measured fastest batch size.

As a result, EdgeRock-4B runs 512 concurrent streams at once, with output throughput of 12124 tokens/s.

The source model Qwen3.5-4B's fastest configuration is batch size 64, at only 1889 tokens/s; Gemma 4 E4B, MiniCPM5-2B, and LFM2.5-2.6B are all between 1700 and 2200 tokens/s.

Why is the gap so large? The math is actually easy.

After running 32K input plus 32K output, Qwen3.5-4B's KV Cache is 2 GiB per sequence.

A single 80GB GPU fills up after a few dozen concurrent sequences, and more requests can only queue.

When EdgeRock-4B runs in batches, the total history state each sequence needs to store is fixed at 97.76 MiB, independent of length.

For comparison: if Qwen3.5-4B were to also run all 512 sequences simultaneously, the KV Cache at generation end alone would be 1 TiB, roughly equal to the VRAM of 13 80GB GPUs.

EdgeRock fits it all on one card, with a peak of 61.26 GiB, including model weights.

Now look at scalability.

Across 16 workloads combining 4K to 32K input and output in pairs, with batch size increasing from 32 all the way to 512, weighted decode throughput rises from 4128 to 12497 tokens/s, and peak VRAM never hits the single-card limit.

Single requests aren't slow either.

Models whose KV Cache grows with reasoning share a common problem: the more they write, the slower they get.

Because for each new token generated, they have to look back over an ever-growing KV Cache.

EdgeRock's computation per token does not grow with context.

After 128K input and then generating 128K tokens, its speed stays steady at 233.7 tokens/s from start to finish;

Qwen3.5-4B, meanwhile, drops from around 169 all the way down to 151.3.

Throughput several times higher means, from another angle, cost several times lower.

Same card, same workload, generating the same number of tokens:

If billed by GPU time, the compute cost per token also drops by the same ratio.

Averaged across the four input lengths, EdgeRock's throughput is 4.94 times that of Qwen3.5-4B and 7.18 times that of LFM2.5-2.6B.

More important is the trend: the longer the context, the bigger the gap.

Qwen3.5-4B's KV Cache grows more and more with length, while EdgeRock's does not grow.

Beyond "saving," fixed memory also brings a benefit that is easy to overlook: determinism.

How much memory each request needs can be calculated clearly before inference begins.

In an era when memory is increasingly precious, this calculation will only become more valuable.

Seeing this, knowledgeable readers may ask: constant memory was already achieved long ago by RNNs and Mamba, so what's so remarkable?

The remarkable part is that while saving memory, reasoning capability remains.

The team compared EdgeRock-4B with 16 bounded-state models, covering lines such as GDN, RWKV, and Mamba, three of which are at the 14B level.

On AIME 2024–2026, EdgeRock-4B averages a single-pass accuracy of 82.9%; the strongest in the control group, RWKV-7 G1j 13.3B, reaches only 18.8%, and AHN-GDN Qwen2.5 14B reaches 11.4%.

Among 29 text benchmarks, EdgeRock-4B ranks first among bounded-state models on 24 of them; its equally weighted average across the 29 is 23.8 percentage points higher than the strongest baseline on this metric, AHN-GDN Qwen2.5 14B.

Compared with the source model, its capabilities are also largely preserved:

This result is already on par with leading small-size models whose memory grows with reasoning.

So how does EdgeRock achieve this?

The core is a new memory mechanism, Rock Memory Anchors (RMA).

Its design draws inspiration from rock anchors: fixing in place the key information needed for reasoning.

Just as a climber drives rock anchors into the cliff face so that every subsequent step has a reliable pivot, RMA "pins" the truly important details encountered during reasoning into a fixed-capacity anchor library, to be read directly when needed.

Specifically, EdgeRock stores the history of each sequence in three parts, each with its own role:

The anchor selection process works like this:

The writer generates a candidate record for each token, and a scoring network scores it;

Every time 256 candidates accumulate, they are compared with the existing records in the anchor library, and only the 1024 highest-scoring ones are kept, with the rest eliminated.

The four KV groups each do this independently.

The retained anchors are independent of one another and their content does not change. So even if their source has long since slid out of the local window, later reasoning can still read them directly.

Put simply, what the model learns is not to "remember everything," but to "remember only what is worth remembering."

The local window plus the anchor library fixes the attention KV of each sequence at 72 MiB. This is the source of the non-growing memory.

Architecturally, EdgeRock-4B is built by modifying Qwen3.5-4B: it retains 24 Gated DeltaNet layers, replaces the 8 full-attention layers with sliding-window attention using a 1024 window, and adds an independent RMA module.

The entire model is converted from Qwen3.5-4B in stages, with no need for pretraining from scratch.

So do the anchors actually work?

The team ran an experiment: during inference, all 8 RMA modules were turned off, while the recurrent layers and local attention worked as usual.

On 256 long-context segments of 16K to 130K tokens, with 4 questions asked per segment, covering retrieval, state updates, cross-document information combination, and variable binding:

Long-distance recall dropped to almost zero. This shows that EdgeRock indeed relies on anchors to read distant information and answer.

Moreover, each anchor records its source position, so you can directly see what the model chose to remember.

EdgeRock was jointly completed by Yanxin Matrix, under Yanshan Technology, together with universities including HKUST and CMU. The model is now open source and available for commercial use.

The team revealed that more parameter-scale EdgeRock models are on the way.

For robots that need to run for long periods, this mechanism also has considerable potential.

After all, a robot perceives the environment while planning actions, and tasks run for hours or even longer; it simply cannot let the history cache expand without bound.

If memory usage during inference can be calculated clearly in advance, on-device deployment can also save a lot of trouble.

As for whether memory prices will keep falling, leave that to the market to answer.

But at least one thing has already changed: no matter how long the model thinks, memory will no longer expand.