InternLM 发布 RNG-Bench：评测多模态大模型在可控非马尔可夫游戏中的表现

InternLM; RNGBench

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork0 (0); Star42 (42)You must be signed in to star a repository

Repository files navigation

README

*Equal contribution †Corresponding authors

[2026.7.7] Added Seed-2.1-Pro to the Main Results — tops single-player Matching Pairs (10×10 image, 64.6%) and ranks #2 on 3D Maze (13×13, GS 32.0%). Results table and project page updated; [2026.6.23] RNG-Bench is on 🤗 Hugging Face: Paper · Dataset (Game Trajectories) · Collection; [2026.6.18] RNG-Bench code and paper released! Check out 📃 arXiv:2606.19338 🎉

Many real decisions hinge on something no longer on screen — a card seen a few turns ago, a corridor already walked. We call this the Non-Markov regime: the current observation is not a sufficient statistic, so a model must reconstruct the relevant hidden state from its history before it acts, and a single recall error changes what it sees next. RNG-Bench isolates this remember-to-act ability in closed loop, under controlled difficulty, with two complementary games.

Two complementary games

🎴 Matching Pairs — static, categorical hidden state. Card identities are revealed for a single turn and must later be recalled by location; 🧭 3D Maze — dynamic, spatial hidden state. Egocentric first-person views must be assembled into a map to reach the goal

Both run under one harness and one strict parser, so a score drop reflects belief-state tracking rather than rule misunderstanding or action formatting.

Highlights

Remember-to-act, in closed loop. The model acts every turn on observations that have since disappeared, and a recall error reshapes the next observation — unlike memory benchmarks that ask a single post-hoc question; Controlled difficulty. Independently vary hidden-state scale (grid / map size), visual pattern, and observation modality (text vs. image), with everything else held fixed; Duel protocol. Two models alternate on the same board, cancelling instance variance and rewarding exploitation of opponent-revealed cards; Long-context stress test. The hardest configurations reach ~128K tokens and ~350 image inputs per episode, and scale further with size; Far from saturation. The best 10×10 image Matching Pairs score is 64.6% (Seed-2.1-Pro); the best 13×13 maze success rate is 50% (Gemini-3.1-Pro)

Main Results

No frontier system is close to saturation.

Single-player. Two separate tables, one per game. Best per column in bold.

Matching Pairs (10×10, image, noise theme) — Score% = fraction of matched pairs; Resp./Score = responses per matched pair; PF/IA = parse-failure / invalid-action rates.

3D Maze (13×13, no minimap, mean optimal path 60 steps) — GS% = aggregate score (success rate, efficiency, exploration); Eff. is over successful episodes only.

Duel — Matching Pairs, image (poker), each model plays 16 games vs. the other four (both player orders, two seeds). The ranking diverges from single-player: Gemini-3.1-Pro wins every matchup, exploiting cards revealed by the opponent.

Key findings. Performance drops sharply with scale (Qwen3.5-397B: 90.6% → 0.7% from 4×4 to 12×12). Vision is the bottleneck, not history length — Qwen3.5-397B and Kimi-K2.5 solve Matching Pairs perfectly in text but fall to 38.3% / 43.3% under noise-pattern images. The textual action trace is load-bearing — removing it collapses GPT-5.4 from 62.3% to 15.3% even though every flip is visible in the board image. And there is large headroom: an optimal policy needs only 3.24 responses per matched pair vs. 7.9 for the best model. Full ablations are in the paper.

What's in this repo

Both games share one model registry (model_presets.py) and one eval harness (framework/); adding a model in either takes effect everywhere. The data-generation engine used for the training experiments below is not part of this release yet.

Quick Start

Install

Configure an OpenAI-compatible endpoint

Both games read API keys/endpoints from a single repo-root .env (loaded automatically by every entry point). Copy the template and fill in what you use:

A model preset only needs whichever key its endpoint uses — you do not need all of them. The endpoints in model_presets.py are illustrative placeholders (localhost / example hosts); point them at your own deployment.

Add or change a model

All models live in the repo-root model_presets.py, shared by both games. A preset maps a short name → endpoint + sampling. The minimal form points at your default .env endpoint:

Then pass --model my-model to any mode. See the existing entries for vLLM/ lmdeploy, Ark, and gateway examples.

Run Matching Pairs

Each run writes game.json (trajectory) and images/round_*.png (per-round renders) under results_demo/ / / / /seed_ /.

Run 3D Maze

Sweeps

Each game ships example launchers under its scripts/ directory. They build a model × config × seed matrix and run it in parallel; everything is overridable via env vars, and PARALLEL (a thin wrapper over xargs -P) is the concurrency knob — set it to whatever your API rate limit allows.

See each game's README.md for the full flag reference.

Citation

License

Code is released under the MIT License (see LICENSE). The card-asset themes under 1_matching_pairs_new/assets/ retain their original licenses (see assets/ /LICENSE where applicable).

Acknowledgements

We thank the maintainers of LLaMA-Factory, vLLM, and the open-weight model teams whose checkpoints we evaluated.