Baidu Unlimited OCR 2 open-sourced, then shared by paid members

Baidu open-sources Unlimited OCR 2: ultra-long document parsing, with attention cost kept from growing with page count as much as possible Unlimited OCR 2 is an open-source OCR / layout parsing model aimed at documents of "dozens to hundreds of pages," with a 3B/A0.5B MoE model architecture: it uses SR-SWA to compress text–vision attention into a fixed-size working reference set, then combines this with page-chunked decoding so that ultra-long documents can be processed within a limited context, and it claims that step-by-step decoding latency and GPU memory barely grow as documents get longer. Open-source model: https://t.co/Z8uP95MZ1Y What changed relative to v1 v1 (R-SWA) could already do multi-page end-to-end parsing, but the paper states that input was roughly capped at 40 pages, and for that reason it was not included in MarathonDocBench. v2 upgrades the decoder attention to SR-SWA (Sparse Reference Sliding Window Attention) and adds Unlimited-Horizon Decoding; the backbone remains the DeepEncoder + MoE decoder line, retrained from the v1 weights. The slogan moves from "long-range" to "unlimited-range." What SR-SWA is doing (per the technical report) Long documents make text–vision attention very expensive. R-SWA kept local interactions in check, but distant visual tokens still grew with page count. SR-SWA's approach is to compress interactions into a fixed-capacity working reference (the README gives 512 tokens as an example). Roughly two steps: Page Gate first narrows candidates by page; Token Sieve then summarizes and scores chunks, using α-entmax sparse routing to keep only the top-k distant chunks plus the local window. On the decoding side, it splits into chunks by page, overlaps page boundaries by 1 page, and reuses the KV tail to avoid GPU memory blowing up at ultra-long lengths. The author claims that at the hundred-page scale, step-by-step speed is about 7.0× that of Full Attention and about 5.6× that of R-SWA.

View original

百度开源 Unlimited OCR 2：超长文档解析，注意力开销尽量不随页数涨 Unlimited OCR 2 是面向「几十到几百页」文档的开源 OCR / 版面解析模型，3B/A0.5B 的 MoE 模型架构：用 SR-SWA 把文本–视觉注意力压到固定大小的工作参考集，再配合按页分块解码，让超长文档能在有限上下文里跑完，并声称逐步解码的延迟和显存几乎不随文档变长而涨。 开源模型：https://t.co/Z8uP95MZ1Y 相对 v1 改了什么 v1（R-SWA）已能做多页端到端解析，但论文写明输入大约卡在 40 页，也因此没上 MarathonDocBench。v2 把解码器注意力升到 SR-SWA（Sparse Reference Sliding Window Attention），并加上 Unlimited-Horizon Decoding；骨干仍是 DeepEncoder + MoE 解码器一路，从 v1 权重再训。口号从「长程」推到「不限程」。 SR-SWA 在做什么（技术报告口径） 长文档会让文本–视觉注意力很贵。R-SWA 管住了局部交互，但远端视觉 token 仍会随页数涨。SR-SWA 的做法是：把交互压进固定容量的工作参考（README 举例 512 tokens）。大致两步：Page Gate 先按页收窄候选；Token Sieve 再对块做摘要打分，用 α-entmax 稀疏路由，只保留 top-k 远端块加局部窗口。解码侧按页切块，页边界重叠 1 页并复用 KV 尾部，避免超长时显存爆掉。作者称百页量级上，逐步速度相对 Full Attention 约 7.0×、相对 R-SWA 约 5.6×。