InternLM 发布 InternLumina-U2 统一多模态模型推理代码与昇腾权重

InternLM; InternLumina-U2

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork5 (5); Star68 (68)You must be signed in to star a repository

Repository files navigation

[2026-09] 🏋️ We release the Ascend-trained model weights on Hugging Face — see Model Weights; [2026-08] 🚀 We release the inference code (this repository). Model weights, the training code and the tech report are coming soon — see the Roadmap

🌟 Introduction

Intern Lumina U2 is a unified multimodal model that brings language, image, video and 3D into a single framework, covering text QA, text-to-image generation, image understanding, image editing, video understanding and 3D understanding with one model. It is a 16B-parameter MoE with 1B active parameters (16B-A1B), pairing an efficient sparse backbone with an 8-codebook fully-discrete visual representation built on AToken.

Discrete unified models (e.g. MMaDA, Lumina-DiMOO) showed that once language and vision share a discrete token interface, understanding and generation no longer need separate model stacks. But going from "can be unified" to "high-performance unified" has been blocked by how visual content is represented and predicted: a single codebook caps the information a visual token can carry, while discrete–continuous hybrids split understanding and generation across different representations and decoding paths.

Intern Lumina U2 explores a new path: fully-discrete multi-codebook modeling.

✨ Highlights

1. One model for omni-visual understanding, image generation and editing. Text QA, T2I generation, image understanding (VQA / OCR / chart / math reasoning), instruction-based image editing, video understanding and 3D understanding all run through the same backbone and the same discrete token interface.

2. Multi-codebook visual representation. The AToken tokenizer describes each visual position with 8 complementary codebooks, expanding the encoding space and preserving local texture, embedded text, geometry and high-frequency detail — without inflating sequence length. On the input side, each codebook keeps its own embedding; the 8 per-position embeddings are concatenated and projected into a single backbone token. On the output side, prediction is spatially parallel, codebook-depth autoregressive: the dLLM backbone denoises many masked spatial positions in parallel, and a multi-codebook AR head predicts the 8 codes of each position in codebook order.

(a) Input-side design: per-codebook embeddings are concatenated and projected into one backbone token.

(b) Output-side design: spatially parallel denoising with a codebook-depth autoregressive head.

3. Architecture built on LLaDA-2.0 MoE. The language backbone is the LLaDA-2.0 MoE diffusion language model(16B-A1B). Its block-diffusion objective, sparse-expert capacity and language competence are extended to unified visual tasks through joint multi-task training.

4. Dual hardware ecosystems. Intern Lumina U2 is adapted to both NVIDIA GPUs and Ascend NPUs across training, evaluation and inference, with operator-level numerical alignment between the two backends and an end-to-end pipeline that switches smoothly between them. On Ascend, systematic optimization — fused operators (e.g. FLA and GMM) that relieve the memory-access and kernel-launch bottlenecks of the sparse MoE architecture and long-sequence training, HSDP sharding tuned to balance communication cost against memory footprint, and gradient checkpointing to raise the maximum per-device sequence length — improves training efficiency on a thousand-card Ascend cluster by 1.85x.

🏆 Benchmarks

Preliminary, partial results. Full comparison tables will appear in the upcoming technical report.

Intern Lumina U2 surpasses unified models such as InternVL-U, LLaDA2.0-Uni, Show-o2 and Lumina-DiMOO on fine-grained understanding and generation benchmarks, and outperforms the understanding-only InternVL3-8B on MathVision and DynaMath.

📦 Release Scope

This repository contains the inference code only. The Ascend-trained checkpoint is available now (see Model Weights); the NVIDIA-trained checkpoint, the training code, and the tech report will be released subsequently — see the Roadmap.

🛠️ Installation

Tested environment: Python 3.11, PyTorch 2.6.0 + CUDA 12.4, flash-attn 2.7.2. The LLM loader pins the backbone to PyTorch SDPA, but the AToken visual tokenizer requires FlashAttention.

Notes:

Importing the model code requires a visible GPU (attention/MoE kernels initialize at import time), so run everything on a GPU machine; liger-kernel and xformers are not required. The LLM backbone runs on PyTorch SDPA; flash-attn is needed only by the AToken visual tokenizer (i.e. every task that encodes or decodes images/video/3D — plain text generation works without it); Object-storage packages (petrel-oss-sdk / boto3) are optional — local token files need neither; Precomputed .pkl/.pt token files are deserialized with PyTorch's trusted-payload mode for legacy NumPy payloads; load only files from a trusted source. To create such files yourself, see Tokenizer Utilities; The root driver is intentionally single-process (INFER_NPROC_PER_NODE=1); run one request per process. It does not partition a batch across GPUs; The root driver changes into the repository root before launching Python, so relative checkpoint, asset, token, and output paths are resolved relative to that root. Set LUMINA_ROOT=/path/to/InternLumina-U2 when invoking the driver from another directory; direct Python entrypoints should receive absolute paths or be run from the repository root; trimesh is installed by requirements.txt because the bundled GLB/GLTF encoder and the mesh fallback need it. plyfile and diff-gaussian-rasterization are optional extensions for external Gaussian-splat assets

🖥️ Ascend NPU

Ascend NPU support lives on the ascend-npu-support branch rather than on main. It runs the LLM backbone on the torch_npu fused attention kernel and routes fused MoE through the NPU grouped matmul, so it needs CANN and a matching torch_npu build in place of the CUDA toolchain and flash-attn above.

📥 Model Weights

Weights are hosted on Hugging Face: internlm/InternLumina-U2. The repo holds checkpoints of the same architecture trained on two different hardware stacks, each in its own subfolder — download whichever matches how you plan to run inference.

Each subfolder is a ready-to-use split HF checkpoint (config.json + safetensors + configuration_internluminau2.py / modeling_internluminau2.py / tokenbridge.py / parallel_plan.py), directly usable as CHECKPOINT below.

🎨 Inference

All inference goes through one driver (infer_1024_sft.sh) with one section per task. It expects a split HF checkpoint directory (e.g. ckpt/ascend from Model Weights above), and — for anything that encodes or decodes images — the AToken tokenizer weights at atoken_inference/checkpoints/atoken-sod.pt (or ATOKEN_MODEL_PATH=...). The model asset directory must keep config.json, configuration_internluminau2.py, modeling_internluminau2.py, and tokenbridge.py together when using Transformers AutoModel*. The system prompt is part of the checkpoint's training contract: for strict reproduction, override the corresponding *_SYSTEM_PROMPT variable with the exact prompt used to train that checkpoint. Existing outputs are overwritten by default; pass --skip_existing to a Python entrypoint only when the full request is unchanged.

The underlying Python entrypoints (infer_t2i_sft.py, infer_mmu_sft.py, infer_video_sft.py, infer_3d_sft.py) can also be called directly; run any of them with --help. The model's low-level generate is a block-diffusion API, so use these task entrypoints instead of the standard Transformers generate(max_length=...) interface.

1. Text Generation

2. Text-to-Image Generation

T2I_SIZE (HxW, default 1024x1024), T2I_CFG (3.0), T2I_TEMP (1.0), T2I_TIMESTEPS (96).

3. Image Understanding

UNDER_MAX_EDGE caps the encode resolution (default 512); leave UNDER_PROMPT unset for captioning; omit UNDER_IMAGE to run the shipped demo images. Decode knobs: UNDER_GEN_LEN (4096), UNDER_BLOCK_LEN (32), UNDER_TIMESTEPS (32).

Natural image (VQA / captioning):

Chart / table / document QA (same entrypoint, denser input):

4. Image Editing

The raw source image is encoded on the fly; defaults run the shipped sample (imgs/edit_source.png + a charcoal-sketch instruction). EDIT_MAX_EDGE caps the encode resolution; EDIT_CFG (1), EDIT_TIMESTEPS (64).

For dual CFG, set both variables together (and omit EDIT_CFG): EDIT_CFG_TEXT=0.5 EDIT_CFG_IMG=0.5.

5. Video Understanding

Defaults run the shipped sample clip (imgs/demo_video.mp4); VIDEO_PROMPT sets your question. VIDEO_NUM_FRAMES (64 sampled frames), VIDEO_MAX_SIZE (448 long-edge per frame), VIDEO_GEN_LEN (1024), VIDEO_BLOCK_LEN (32), VIDEO_TIMESTEPS (32).

6. 3D Understanding

Defaults run the shipped sample asset (imgs/demo_3d.glb); PROMPT_3D sets your question. Decode knobs: GEN_LEN_3D (128), BLOCK_LEN_3D (32), TIMESTEPS_3D (32).

GLB/GLTF understanding encodes the asset exactly as in training — the bundled atoken_inference/glb_encode module drives Blender to render 64 color/depth views, back-projects them into a voxel grid, and feeds AToken. Install Blender once — infer_3d_sft.py then finds it automatically:

A Blender already on PATH or pointed to by BLENDER_BIN also works. On minimal containers a few X/GL system libraries may be missing; the setup script verifies headless startup and prints the exact apt-get line (or set BLENDER_LIB_DIR to a directory of the missing .so files if you cannot install system packages). Precomputed 3D tokens (--token_path) skip encoding entirely and never need Blender.

🧰 Tokenizer Utilities

tools/tokenize_image.py pre-encodes images into the token files the inference entrypoints accept (--input_token_paths for editing, --token_path for understanding) — useful to encode once and reuse across runs, or to inspect what the model actually "sees":

For 3D assets the equivalent offline encoder is atoken_inference/glb_encode/encode_glb_tokens.py (--glb ... --output ...; requires Blender, see the 3D note above).

🗺️ Roadmap

Inference code for T2I / editing / image, video and 3D understanding; Model weights (Ascend); NVIDIA checkpoint coming soon; Training code; Ascend NPU training & inference stack; Tech report

🙏 Acknowledgements

This project builds on the excellent work of:

LLaDA-2.0 — the MoE diffusion language model backbone; AToken (Apple ml-atoken) — the multi-codebook visual tokenizer used for all visual modalities; atoken_inference/ is derived from it; Lumina-DiMOO & LLaDA2.0-Uni — pioneering discrete-diffusion unified modeling; VeOmni — vendored from upstream commit 600fe6d with local compatibility patches for this model

📜 License

This project is released under the Apache License 2.0. The bundled VeOmni/ directory retains its upstream Apache-2.0 license. atoken_inference/ is derived from Apple's ml-atoken and is redistributed under Apple's original license terms (atoken_inference/LICENSE).

📖 Citation