Kyutai open-sources CADRE video autoencoder, reducing world model action latency to under 230 ms
Kyutai releases the open-source causal diffusion representation autoencoder CADRE, aimed at interactive world models, running at 20 fps on 1 H100 with action latency below 230 ms. CADRE contains 7 autoencoders of about 476M parameters each, paired with a frozen 86M DINOv3 ViT-B/16; the world model has 1.2B parameters, and the video resolution is 288×512. The driving-scenario FVD is 162.6, compared with 599.7 for Wan 2.2; however, the action-following ARR for robotic manipulation is 0.45, lower than 0.85 for Wan 2.2. The code uses the MIT license, the Hugging Face weights are CC BY-NC 4.0, and commercial authorization is not included.
Why read this
AI score 58
The original text uses the same world model to horizontally compare multiple video autoencoders, breaking down the relationship between action latency and compression rate into a comparable table.
Topics
Body
Add as a preferredsource on Google
Kyutai has released CADRE, open causal diffusion representation autoencoders for interactive world models. The Kyutai CADRE video autoencoder targets one problem: action lag. Many world models borrow autoencoders from text-to-video generators. Those autoencoders pack many frames into one latent, so every action waits. Per the research paper, CADRE world models run at 20 fps on 1 H100 and react within 230 ms.
TL;DR
Size: 7 autoencoders, each about 476M trained parameters plus a frozen 86M DINOv3 ViT-B/16. The world model is 1.2B parameters. Video runs at 288×512, 20 fps.
Runs on: 1 H100 (bf16, batch 1). Quantized variants: Not disclosed.
Performance: CADRE forms most of the latency-quality frontier. It is as fast and better in 42 of 48 comparisons with MIRA, Wan 2.2, Cosmos and LTX-2.
Best: Driving FVD of 162.6 (CADRE-2×16×16, 1 step) against 599.7 for Wan 2.2.
Worst: Robot manipulation action following (ARR) of 0.45 for CADRE-2×32×32, against 0.85 for Wan 2.2.
What is CADRE?
CADRE is a family of video autoencoders from Kyutai built for action-conditioned world models. A world model generates in the autoencoder’s latent space. So the autoencoder’s compression sets both cost per frame and action delay.
TEXTTABLELAYOUTIs your PDF parser actually getting it right?omni_parse_bench · scores text, tables & layout · open sourceStar on GitHub→Sponsored
Compression is written τ×s×s. τ is frames per latent, and s is the spatial factor. The research gives the latency as τ/f plus generation and decode time. At τ = 8 and 20 fps, the chunk alone adds 400 ms. Humans notice their own movement lagging beyond about 230 ms.
How does CADRE work?
The encoder reuses the MIRA recipe. A frozen DINOv3 extracts per-patch features from each frame. One trained 3D convolution then compresses them into a 64-channel latent. Only this bottleneck changes between the 7 members.
The decoder is a causal diffusion transformer with 24 blocks. It streams chunk by chunk. Progressive self-distillation teaches it to decode in 1 or 2 steps. This happens in a single training stage, without an adversarial loss.
The autoencoders train on 23 public video sources, about 76k hours. The mix spans web, robot, driving and game video.
How was CADRE evaluated?
Kyutai trained the same 1.2B causal diffusion transformer on each autoencoder. It used 4 environments: robot manipulation (ABC), human manipulation (EgoDex), driving (PhysicalAI AV) and indoor navigation (RealEstate10K). Each model sees 2 s of real video, then generates 4.8 s.
Scores cover image quality (FID), motion quality (FVD) and action following (ARR). ARR uses a frozen inverse dynamics probe. An ARR of 1 matches real video.
What do the benchmarks show?
On one H100, CADRE-2×32×32 reacts in 185 ms and sustains 23.4 fps with 1 decoding step. The Wan 2.2 autoencoder needs 317 ms, LTX-2 492 ms and Cosmos 605 ms. DC-AE is faster at 118 ms but frame-by-frame.
The gap is widest with a moving camera. In driving, Wan 2.2 FVD reaches 599.7 and LTX-2 913.3. In indoor navigation, LTX-2 ARR falls to 0.04. Against the retrained MIRA decoder, the 1-step diffusion decoder lowers FVD by 1.4 to 2.5×.
What are the 4 findings?
Decode by diffusion: at strong compression, perceptual regression blurs, while diffusion samples 1 sharp video.
Compress time in the autoencoder: patching inside the world model loses detail. At 8 frames, ARR drops from 0.72 to 0.56.
Camera motion decides compression: still cameras tolerate temporal compression. Moving cameras favor fewer frames and coarser spatial tokens.
Coarse tokens hurt motion first: in robot manipulation, 32× spatial compression multiplies FVD by 2.4 over 16×. FID barely moves.
Comparison: CADRE vs other video autoencoders
FeatureCADRE-2×32×32CADRE-2×16×16MIRA recipeWan 2.2 VAECosmos CV8x8x8LTX-2 VAE
Compression (τ×s×s)2×32×322×16×162×32×324×16×168×8×88×32×32 (paper)
Tokens per 16 frames11524608115223044608288
DecoderCausal diffusion, 1 to 2 stepsCausal diffusion, 1 to 2 stepsPerceptual regressionCausalCausalNon-causal, windowed
Action latency, 1 H100185 ms216 ms175 ms317 ms605 ms492 ms
Max fps, 1 H10023.417.326.634.239.087.4
FVD, driving (lower is better)171.3162.6311.6599.7452.5913.3
FVD, robot manipulation105.047.6267.152.939.569.6
ARR, robot manipulation (higher is better)0.450.730.390.850.780.25
Reconstruction PSNR (OpenVid-HD)22.8625.1224.4435.0231.3928.24
Weights licenseCC BY-NC 4.0 + DINOv3 LicenseCC BY-NC 4.0 + DINOv3 LicenseNot disclosedApache 2.0NVIDIA Open Model LicenseLTX-2 Community License
Latency, fpsA, FVD, ARR and PSNR: CADRE paper, Tables 4, 6 and 7. CADRE uses 1 decoding step. All figures are self-reported by Kyutai.
Where does CADRE fall short?
CADRE does not win everywhere. In robot manipulation, Cosmos posts an FVD of 39.5 against 47.6 for the best 1-step CADRE. At the fast end, DC-AE beats CADRE-1×48×48 in manipulation. Wan 2.2 also reconstructs held-out video far better, at 35.02 dB PSNR.
Kyutai team argues reconstruction does not predict rollouts. Wan 2.2 ranks last on indoor rollout FVD.
Long latents leave visible seams between chunks. At τ = 8, boundary excess reaches 1.6 to 1.8 across environments. CADRE-4×24×24 is an outlier in navigation, with FVD 403 in driving. Each world model covers one environment at 288×512. Whether findings transfer to generalist models remains open.
How can developers use CADRE?
The code is MIT licensed. Weights on Hugging Face are CC BY-NC 4.0, so commercial use is excluded. The release covers 7 autoencoders, 4 longer-trained multi-resolution versions and 4 world models on CADRE-2×16×16. A live demo lets you drive them by keyboard.
Key Takeaways
7 open autoencoders, one bottleneck apart, isolate compression as the only variable.
World models on CADRE run at 20 fps on 1 H100, under 230 ms lag.
Driving FVD drops to 162.6, against 599.7 for Wan 2.2.
Still cameras favor temporal compression; moving cameras favor spatial compression.
Weights are non-commercial (CC BY-NC 4.0); code is MIT.
Michal Sutter+ postsBio ⮌
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Michal SutterExa 3.0 Search API Tops Perplexity, Brave and Parallel on 8 Benchmarks
$uv pip install omni-parse-bench✓text✓tables✓layout→ scored per pageStar on GitHub→Sponsored
RELATED ARTICLESMORE FROM AUTHOR
Moonshot AI Releases Kimi K3.1 Pro: A 2.8T Total/104B Active Params (MoE) Model with 1M Context
You.com Launches Inference Router: 12 Open-Weight Models With 99.9% Uptime SLA
Gradium Open-Sources Phonon, a 31M-Parameter On-Device TTS Model Under Apache 2.0