蚂蚁 inclusionAI 开源 LLaDA2.0-Uni 统一多模态模型

inclusionAI; LLaDA2.0-Uni

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork43 (43); Star700 (700)You must be signed in to star a repository

LLaDA2.0-Uni: Understanding and Generation the World.

Repository files navigation

README

AGI Research Center, Inclusion AI

[2026-05-29] 📣 SGLang Omni support is ready. See cookbook for installation and usage; [2026-05-12] 🖥️ We release ComfyUI and Diffusers support. See apps for installation and usage; [2026-05-06] ⚡ We release the FP8 quantized versions on HuggingFace and ModelScope; [2026-04-23] 🎉 We release the initial version of LLada2.0-Uni, including:

🎯 Model Checkpoints on HuggingFace!; 🎯 Text-to-Image (w/ thinking mode) Inference Code!; 🎯 Image Understanding Inference Code!; 🎯 Image Editing Inference code!; 🎯 SPRINT Acceleration for dLLM Backbone!

📝 TODO

Quantized model; Diffusers support; ComfyUI support; SGLang support

📚 Model Introduction

We introduce LLaDA2.0-Uni, a unified dLLM-based Mixture-of-Experts (MoE) model that seamlessly integrates multimodal understanding and generation.

Architectural Innovations

Unified dLLM-MoE Backbone: Built on LLaDA 2.0, it unifies multimodal understanding and generation into a simple Mask Token Prediction paradigm; Discrete Semantic Tokenizer: Utilizes SigLIP-VQ to convert visual inputs into discrete semantic tokens, significantly enhancing multimodal understanding; Efficient Diffusion Decoder: Pairs discrete tokens with a specialized diffusion decoder for high-fidelity generation, enabling rapid 8-step inference via distillation

Core Capabilities

Top-Tier Understanding & Generation: Matches dedicated VLMs in answering visual questions and understanding documents, while also generating highly detailed images; Flexible Image Editing: Supports single or multi-reference editing. It enables precise modifications while perfectly preserving original details; Interleaved Generation & Reasoning: Empowered by unified discrete representations, it effortlessly handles complex interleaved generation and unlocks advanced interleaved reasoning

1. Create a conda environment

2. Install PyTorch (CUDA 12.4)

3. Install Flash Attention 2 (required for efficient inference)

4. Install remaining dependencies

💡 Faster decoding — Use the decoder-turbo (distilled decoder) for ~10× faster image decoding (8 steps instead of 50) with minimal quality loss:

🌟 Text-to-Image Generation with Thinking

SPRINT accelerates inference by combining KV cache reuse, adaptive unmasking, and threshold-based batch acceptance:

KV Cache Reuse & Pruning: The prefix KV cache is computed once during warmup steps, then optionally pruned by importance scores (blending KV attention importance with token confidence). Subsequent denoising steps reuse the cached prefix, significantly reducing computation. Per-modality keep ratios (image_keep_ratio, text_keep_ratio) allow fine-grained control — e.g., retaining all image/text tokens for quality while still benefiting from cache reuse; Adaptive Unmasking: Instead of unmasking a fixed number of tokens per step, Sprint dynamically decides how many tokens to reveal based on model confidence. At each step, it computes confidence scores (via strategies like low_confidence, top_k_margin, or neg_entropy) and transfers the top-k most confident tokens, where k is adaptively set as ceil(remaining_masked / steps_left). This allows easy positions to be resolved quickly while concentrating compute on harder tokens; Batch Acceptance: On top of adaptive scheduling, all tokens whose probability exceeds threshold are accepted in batch, further reducing the number of denoising iterations needed

Image Understanding with Sprint:

Text-to-Image with Sprint:

Note

Sprint is supported for Simple CFG and no-CFG modes. When using Editing CFG (three-way guidance with cfg_text_scale / cfg_image_scale), Sprint automatically falls back to baseline.

🌟 Using CLI Scripts

🖥️ ComfyUI Support

We provide native ComfyUI custom nodes for visual, node-based workflows. All three capabilities (text-to-image, image understanding, image editing) are available as drag-and-drop nodes.

Installation

Or use the one-line installer:

Available Nodes

Workflow Examples

For full documentation, see apps/comfyui/README.md.

🚀 SGLang Support

We now support SGLang for high-throughput serving and optimized inference.

For detailed configuration and examples, please refer to cookbook.

⚠️ License

This project is licensed under the terms of the Apache License 2.0.

📖 BibTeX

LLaDA2.0-Uni: Understanding and Generation the World.