Latest commit
History
2 Commits2 CommitsFolders and files
Repository files navigation
Code World Model 2.0: Towards Infinite Controllable Video GenerationInvoker-Render: proxy-conditioned video rendering for offline creation and real-time streaming
Ruoyu Wang1,2,* · Hao Zhong2,3,* · Yanming Yang1,2,* · Jiahao Tian1,2,* · Lin Liu2,*,†
Biao Gong2 · Guoshan Lu2 · Yiding Tian2 · Chunhua Shen3,‡ · Zhenzhong Lan1,2,‡ · Chi Zhang1,‡
1Westlake University 2AGI Research Center, Inclusion AI 3Zhejiang University
*Equal contribution †Project leader ‡Corresponding author
Highlights · Demo · Quick Start · Documentation · Citation · 简体中文
Code World Model 2.0 keeps the world state in an upstream engine and hands rendering to a video model. The two are connected by visual proxies: plain structural videos that fix the camera path, scene layout, and entity motion, and leave appearance to the renderer.
Invoker-Render is that renderer. It takes a structural proxy, one appearance reference image, and text. This repository is the inference codebase for its causal four-step streaming model, built on MiniMax-H3, SGLang, and SGLang-Omni. The method, training data, and full evaluation are in the technical report.
Highlights
- Unbounded streaming, no visual jumps. Each chunk reads a fixed appearance anchor and a bounded window of recent history, so a session runs for as long as proxy frames arrive. The racing demo below is one continuous 30-minute generation.
- Coarse or fine proxies, chosen at run time. One model accepts semantic maps, sparse 3D boxes, subject masks, shaded geometry, and boxes with skeletons, at any level of detail.
- Four steps per chunk. The streaming model is trained with Progressive Forcing, a staged curriculum described in the report.
- Ahead of recent streaming renderers. On 140 held-out rollouts it ranks first on all four judged axes, with an overall Elo of 1343.2 against 1245.3 for SolarRenderer (Causal), 1051.9 for StreamRender-H3, and 359.6 for Streaming CWM.
- Ready to serve. A WebSocket session service on eight GPUs, a file client, and an interactive browser client.
Demo
Every clip shows the input proxy as an inset: the inset fixes how the camera and the objects move, and the model generates materials and lighting.
Long-horizon streaming
One continuous 30-minute generation from a coarse racing proxy (inset, upper left), with no cut or reset. Six two-second excerpts; the timestamp is the position in the take. The full video link is coming soon.
Interactive driving
A gamepad-driven session: the gamepad (inset, lower right) steers the low-poly proxy (inset, lower left), and the rendered car follows. The car, the two cars ahead, and the dusk track stay consistent throughout.
Camera control
Same scene, same first frame, same event; only the camera route changes. Route A is on the left and route B on the right, each with its own proxy inset.
Coastal tunnel. A: a drone stays behind the car through the tunnel and rises after the exit. B: the camera lifts over the headland as the car enters, waits above the exit, and follows the car out.
Rocket launch. A: a fixed ground camera tilts up and pushes in. B: a drone approaches the tower at low altitude, climbs with the rocket, and ends above the tower looking toward the coast.
Grand prix hairpin. A: a broadcast camera on the race-control roof pans and zooms with two cars entering the corner side by side. B: an FPV drone dives from the roof to the track and follows above and behind the overtaking car.
The previews are downsampled excerpts played at 6 FPS; assets/demo/metadata.json lists the source clips. Playback FPS is not inference throughput. The 30-minute racing demo was generated with a reference-guided quality policy that the default service does not enable.
Release Status
- Streaming inference runtime, session service, file client, and browser client
- Deployment guide
- Model weights on Hugging Face
- Homepage
- Technical report on arXiv
- Full 30-minute demo video
- Creation-model inference, training code, and data
Unchecked items have not been published, and the arXiv badge above is a placeholder until the report is. Model weights are hosted on Hugging Face; they and prepared bundles are not included in this repository.
Quick Start
Requirements: Linux, Python 3.11 or 3.12, and NVIDIA CUDA. The default production profile uses eight GPUs on one node. There is no standalone lockfile; provision the environment with the environment guide, then check it from the repository root:
PY=/path/to/serving-env/bin/python "$PY" invoker_render.py check-env
1. Prepare a bundle. Model weights are published on Hugging Face; point the /assets/... paths below at your local copies. Preparation hashes and references those files without copying them. The output directory must not exist yet. See Bundle preparation for each file.
ARTIFACT=/assets/invoker-render/artifact.json BUNDLE=/assets/invoker-render/bundle "$PY" invoker_render.py prepare \ --artifact "$ARTIFACT" \ --cwm-silence /assets/invoker-render/silence.safetensors \ --decoder-config /assets/causal-decoder/config.json \ --decoder-weights /assets/causal-decoder/decoder.safetensors \ --state-reservation-gib 48 --output "$BUNDLE"
2. Start the streaming service. It binds loopback and serves sessions at /v1/session.
"$PY" invoker_render.py serve --model-path "$BUNDLE" --port 18860
3. Render a video from files. Run this in a second terminal.
"$PY" invoker_render.py infer \ --model-path "$BUNDLE" --artifact "$ARTIFACT" \ --uri ws://127.0.0.1:18860/v1/session \ --first-image /inputs/first.png \ --proxy-video /inputs/proxy.mkv \ --prompt-file /inputs/prompt.txt \ --seed 42 --output-dir /outputs/invoker-render-001
The first image is 1344×768 RGB. The proxy video is 24 FPS with 17*N+5 frames, at least 39. The output directory must be new; a completed run writes video.mp4 and inference.json.
4. Open the browser client (optional). It reuses the running service.
"$PY" invoker_render.py web-adapter \ --model-path "$BUNDLE" --artifact "$ARTIFACT" \ --native-uri ws://127.0.0.1:18860/v1/session --port 18861 "$PY" invoker_render.py web \ --backend http://127.0.0.1:18861 --port 18862 --incremental-proxy
Open http://127.0.0.1:18862/ and select a first image that matches the scene.
Documentation
| Task | Guide |
|---|---|
| Configure dependencies and CUDA | Environment |
| Prepare checkpoints and model assets | Model contract · Bundle preparation |
| Choose execution and codec placement | Deployment profiles |
| Run the streaming API and file client | Serving and file inference |
| Use the browser client | Browser deployment · Client guide |
| Check timing and completion records | Output verification |
invoker_render.py Command line: prepare, serve, infer, web-adapter, web, check-env
docs/ Deployment documentation
src/sglang/python/sglang/multimodal_gen/ Native inference runtime
src/sglang/projects/minimax_h3/ Model adapters, tools, configs, browser client, vendored frameworks
Citation
The technical report is coming soon on arXiv. Until then, please cite:
@techreport{wang2026cwm2,
title = {Code World Model 2.0: Towards Infinite Controllable Video Generation},
author = {Wang, Ruoyu and Zhong, Hao and Yang, Yanming and Tian, Jiahao and Liu, Lin and Gong, Biao and Lu, Guoshan and Tian, Yiding and Shen, Chunhua and Lan, Zhenzhong and Zhang, Chi},
institution = {Westlake University; AGI Research Center, Inclusion AI; Zhejiang University},
year = {2026}
}
Acknowledgements and License
Invoker-Render builds on MiniMax-H3, SGLang, SGLang-Omni, and the other vendored open-source components listed in Third-party notices. We thank their authors.
The code written for this repository is released under the Apache License 2.0. Vendored components keep their own licenses, which include Apache-2.0 and CC BY-NC 4.0 (non-commercial) terms; the license file inside each component directory governs that component. Model weights and other separately distributed assets have their own terms. Read the legal notice before use or redistribution.
AI 热点