Baidu Open-Sources ERNIE Unlimited OCR 2, Peak KV Cache Reduced from 47GB to 1.77GB

KV Cache compressed to a constant! A new open-source OCR king: no page limit, cache cut by 96%

Baidu's OCR has been upgraded again, and ERNIE Unlimited OCR 2 has been officially released and open-sourced!

This time, it directly takes on the challenge of reading an entire novel series in one go.

Seven 《Harry Potter》 books, from 《Philosopher's Stone》 to 《Deathly Hallows》, recognized and transcribed continuously, 3600+ pages read all the way to the end.

Switching to the complete 《Three-Body》 series, it can likewise read it all in one go—three books, 1200+ pages, hundreds of thousands of characters, without pausing for breath in between.

However, reading so many pages continuously means the model first has to pass one hurdle: VRAM.

The longer the document, the more intermediate information (KV cache) must be retained during inference—you can think of it as the cheat sheet the model keeps while reading; the more pages, the thicker the cheat sheet. A thick cheat sheet not only takes up space but is also slower to flip through.

In the hundred-page-level single-sequence test in the technical report, ERNIE Unlimited OCR 2 gave a set of comparisons:

Peak KV cache usage dropped from 47GB under standard decoding (keeping the visual caches of all pages in VRAM at the same time) to 1.77GB, a reduction of about 96% at once.

What does 47GB mean? It is larger than the total memory of many laptops. And this is only the KV cache; other overheads such as model weights are counted separately.

Its previous generation, Unlimited OCR, had already gained considerable popularity in the open-source community thanks to its ability to continuously parse multi-page documents: cumulative downloads exceeding 10 million and about 28,600 Stars on GitHub.

From online demos created by the community to support for SGLang inference, adaptation around it has continued to advance.

In the second generation, the upgrade focuses on continuously processing longer documents while reducing the computational and caching burden.

To this end, ERNIE Unlimited OCR 2 proposes the Sparse Reference Sliding Window Attention mechanism, SR-SWA, allowing the model to select relevant pages and visual information on demand and reduce redundant computation.

Together with the continuous decoding mechanism, the model can continuously process ultra-long documents, and the per-step decoding cost does not rise as the document grows longer.

Currently, ERNIE Unlimited OCR 2 has been officially open-sourced on Hugging Face and GitHub.

Open-source address:

Breaking down the core technology: how SR-SWA computes less, and how chunked decoding stores less

How exactly does ERNIE Unlimited OCR 2 manage to continuously read the complete 《Harry Potter》 series—millions of characters and 3600+ pages?

What it draws on is actually the human approach to copying books.

What is the process when a person copies a book by hand? Look at a passage, copy a passage, with the eyes mainly fixed on the page being copied; only when a place requires cross-reference does one turn back to flip through the relevant pages.

Humans figured out book copying earlier than models: no one, while copying 《The Analects》, simultaneously spreads out all the pages of 《The Analects》 in their mind and looks at them together.

But the previous generation, Unlimited OCR, did not do this: although R-SWA had already greatly reduced the KV cache during inference, in the review process the visual information of all input pages still had to participate in attention computation.

The longer the document, the more information is computed, and a considerable portion of it has nothing to do with the few lines currently being recognized—a typical case of "running along for nothing."

△SR-SWA draws on the local-focus approach humans use when copying books

Following the human approach to copying books, ERNIE Unlimited OCR 2 introduces the Sparse Reference Sliding Window Attention mechanism (SR-SWA).

Its filtering has two layers: first filter out the pages that need to be referenced, then pick the truly relevant local information from those filtered pages, keeping irrelevant computation out.

From the overall architecture, ERNIE Unlimited OCR 2 continues the basic design of the previous generation.

△Overall architecture of ERNIE Unlimited OCR 2

The book page image first enters DeepEncoder and is converted into visual Tokens that the model can process—you can think of them as a set of data cards recording page content and layout features.

Then, the language model generates recognition results step by step based on these cards.

The key change occurs inside the language model: the Reference Sliding Window Attention mechanism (R-SWA) used in the previous generation is upgraded to SR-SWA in this generation.

What is shown in the lower left of the architecture diagram is the two-layer filtering process of SR-SWA.

The first layer is Page Gate, which governs "which pages to look at."

Based on the position the model is currently parsing, it narrows the candidate visual information to the relevant page range, keeping irrelevant pages from joining in.

In the book-copying scenario, this is like, when reading a certain page, first finding the few pages related to this page rather than flipping through the entire book from the beginning.

The second layer is Token Sieve, which governs "which lines on this page to look at."

Choosing the right pages is only the first step: even within these pages, not all visual Tokens need to participate in computation. Token Sieve performs another round of filtering, picking out from the pages the visual information blocks more relevant to the current content, in order to capture long-range visual dependencies.

The right side of the architecture diagram expands this process: first use brief representations of each information block to compute relevance, then use TopK to pick out the highest-scoring information blocks in a fixed number.

After picking, the model still looks at the specific visual information in these blocks, rather than only looking at a summary.

The two-layer mechanism works together so that the model can both find the relevant pages within a long document and reduce the amount of visual information it needs to process at each step, thereby improving long-document parsing efficiency.

△Token Sieve details

The filtered visual information also participates in parsing together with the Local Window (local sliding window) in the figure.

This window has a fixed size, holds the most recent context, and shifts forward continuously along with the generation process, responsible for letting the model continue writing from what it has just produced.

The number of information blocks is fixed, and the local window size is also fixed, so the amount of information that needs to be referenced at each step is bounded—no matter how long the document is, this step still involves the same amount of work. There is no situation where "the thicker the book, the harder it is to write each character."

SR-SWA is responsible for computing less at each step; how much information actually needs to be kept in GPU memory is another issue.

After encoding, each page image becomes a set of visual Tokens. The model needs to reference them when generating parsing results, so the corresponding KV cache is stored for reuse in subsequent computation.

If the entire document is read in all at once and the visual caches of all pages are kept in GPU memory, the more pages there are, the more space is occupied.

What is more troublesome: even if these pages do not participate in the current computation at all, as long as the cache is not released, they still occupy GPU memory. This is like copying a book while spreading every page of draft paper on the desk and not throwing away a single page—no matter how big the desk is, it will not be enough.

So, computing less must also be paired with storing less.

To this end, ERNIE Unlimited OCR 2 introduces the Unlimited-Horizon Decoding Paradigm, controlling cache usage through chunked processing.

△Chunked decoding schematic

This mechanism divides the document into page chunks of controllable size and parses them in sequence. Two adjacent chunks overlap by one page; the last page of the previous chunk is also the starting point of the next chunk—a ready-made bookmark.

When entering the next chunk, the model directly reuses the visual KV cache of this overlapping page, while retaining the text cache at the end of the previous output, so that the preceding and following content connect. The remaining visual caches of pages that are no longer needed are released promptly to free up GPU memory.

In this way, the visual cache that needs to be retained in GPU memory is compressed to the scale of the current processing chunk. After finishing one chunk, the information needed for connection is retained, and then the next chunk begins—there is no need for the visual cache of the entire document to occupy space from beginning to end.

*SR-SWA is responsible for reducing computation at each step, and the unlimited decoding mechanism is responsible for controlling cache usage. Only with the two working together can the model continuously parse ultra-long documents. 884-page The Lord of the Rings read in 42 minutes, cache reduced to 1.77GB. Having discussed so much technology, we also wanted to see the effect with our own eyes: directly throw it an 884-page English edition of The Lord of the Rings.

After about 42 minutes and 56 seconds, parsing was complete. 884 pages of input corresponded to 884 pages of output, averaging about 2.9 seconds per page. About the time of one TV episode, and "Middle-earth" completed an entire journey. For such a long document, can recognition quality still keep up when processing reaches the later parts? We specifically spot-checked the main text on pages 600 and 750: the recognized text matched the original. Still being word-for-word accurate on page 750 shows that the model did not "get carried away" earlier on.

Throughout the task, the system also marked 466 cross-page merges. For example, pages 873 to 874 near the end of the book: sentences forcibly cut off by pagination were also reconnected in the output.

The actual test demonstrated the process of continuously parsing the entire book, while the technical report answered three other questions: how much cache is needed, how fast it can be, and what recognition quality is like. On the hundred-page-level benchmark MarathonDocBench, the team compared standard decoding with Unlimited-Horizon Decoding side by side. In single-sequence inference, after adopting chunked processing and boundary cache reuse, peak KV cache usage dropped from 47GB to 1.77GB, a reduction of about 96%.*

The large amount of page information that originally needed to remain in GPU memory simultaneously can now be processed in batches and released promptly. The 1.77GB here refers only to the KV cache; other GPU memory overheads such as model weights are counted separately.

Beyond saving cache, the report also measured performance when processing multiple documents simultaneously.

In another 16-way concurrency test, after adopting this decoding approach, end-to-end throughput increased from 360 to 1315 Tokens/second, about 3.65 times that of standard decoding.

△Comparison of decoding methods: single-sequence peak KV cache and 16-way concurrent throughput

For services that need to batch-process annual reports and technical manuals, both changes have practical significance: with lower GPU memory usage, the same card can fit more work; with concurrency running, more content is output per unit time.

Users also do not need to worry; they still submit the entire document as before, and chunking and connection are completed internally by the model, eliminating the trouble of manually splitting files, submitting in batches, and then stitching them together.

Beyond being faster and more economical, parsing quality must also hold up.

MPDocBench examines the parsing quality of multi-page documents: when facing documents dozens of pages long, the model must both accurately recognize content and restore document structure.

In this test, ERNIE Unlimited OCR 2 achieved a comprehensive score of 84.59.

To further test hundred-page-level documents, the team also built MarathonDocBench. It contains 100 documents, each 103 to 300 pages long. Many paragraphs and tables continue onto the next page, specifically testing whether the model can correctly connect content across pages.

On this longer and trickier benchmark, ERNIE Unlimited OCR 2 achieved a comprehensive score of 77.06. In both evaluations, its comprehensive score was the highest among the models listed in the report.

△From the technical report: multi-page document parsing evaluation results

The change in long-document processing speed is equally evident. In the report's hundred-page-level tests, SR-SWA's per-step decoding speed reached 7.0 times that of Full Attention and 5.6 times that of R-SWA used in the previous generation.

The curves make it more intuitive: as documents grow longer, the per-step decoding time of the first two approaches climbs steadily, while SR-SWA's per-step time is almost a horizontal line.

This means the time the model needs to generate each Token does not noticeably increase as the number of input pages grows—the thicker the book, the same the writing speed.

△Per-step decoding latency under different document lengths

Compared with the previous generation, ERNIE Unlimited OCR 2's per-page comprehensive score is basically on par.

The real gap emerges after documents grow longer: more pages can be processed continuously, cross-page content is recognized more accurately, and the computation and cache overhead of long-document parsing drops accordingly.

As for how everyone uses it...

You can input annual reports of several hundred pages, or a thick technical manual.

Especially for scanned documents: first hand them to the model for continuous parsing and conversion to text, then retrieve information and verify content, which also saves a lot of the hassle of splitting pages and manual stitching.

Of course, from a few pages of material to an entire book, the problems OCR needs to solve are also changing.

In the technical report, the team summarizes the evolution of document parsing into three stages: OCR 1.0, 2.0, and 3.0.

Early OCR was more like a clearly divided assembly line: first locate the text, then recognize its content, and finally organize the layout and output the result. If multi-page parsing was involved, it was usually processed page by page and then the results were stitched together—this is OCR 1.0.

Different modules each performed their own duties, but an error in an earlier step could also propagate all the way to the end—when one link sneezes, everything downstream catches a cold.

By OCR 2.0, vision-language models began generating parsing results directly from page images, reducing handoffs between multiple modules. The first-generation Unlimited OCR extended this end-to-end parsing to documents of dozens of pages.

When documents grow from dozens of pages to hundreds, continuous parsing must both ensure content continuity and control computation and cache overhead.

Through on-demand reference and chunked decoding, ERNIE Unlimited OCR 2 allows the model, when processing more pages, to avoid expanding the computation scope of each step with the number of pages, and to avoid keeping the visual cache of all pages at all times.

The team calls this step OCR 3.0: making continuous processing of an entire document the new technical goal—in one breath, without pagination, unlimited parsing, never crashing.

And moreover, this goal also has an anime version.

The name Unlimited OCR Works itself pays homage to "Unlimited Blade Works" from *Fate/stay night*.

△The homepage of the *Fate/stay night* anime official website

In the original setting, this is a magic called "Reality Marble"; when cast, it unfolds a special space filled with countless swords.

After the previous generation was released, Hugging Face user Snarcy discovered this naming meme and simply adapted the classic incantation into an OCR version.

By the second generation, the team actually included this adaptation in the introduction of the technical report and retained Snarcy's attribution.

So, opening this report discussing attention mechanisms and GPU memory overhead, at the very beginning you will see this incantation:

Pixels are my body, and tokens are my blood.

Pixels are the body, and Tokens are the blood.

I have parsed over a thousand pages.

Therefore, I pray here: Unlimited OCR Works.

From 47GB down to 1.77GB—the line "not limited by KV cache" really was not sung in vain.

Friends who feel this deeply are welcome to like, follow, and share—վ’ᴗ’ ի ❤