InternLM 开源 InstantFusion：为异构图像生成器提供共享潜空间接口
GitHub - InternLM/InstantFusion: An official implementation of "InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators" · GitHub
InternLM 发布 InstantFusion 官方实现，通过共享潜空间将不同扩散模型的潜变量映射对齐，使去噪轨迹可组合。其核心是 Sigma 条件化的潜空间对齐编码器（LAE），训练结合重建、交叉重建、潜空间对齐与去噪速度对齐损失。该框架支持不同步数预算模型间的加速交接、多偏好组合，以及跨模型 OPD（以 Qwen-Image 为冻结教师训练 SD3 LoRA）。
正文
当前显示原文，中文全文译文尚未就绪。
InternLM; InstantFusion
Repository navigation
NotificationsYou must be signed in to change notification settings; Fork0 (0); Star21 (21)You must be signed in to star a repository
An official implementation of "InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators"
Repository files navigation
README
Composing diffusion models through a shared latent space
Overview
InstantFusion maps the latents of different diffusion models into a shared latent space, so their denoising trajectories can be aligned and composed.
Latent alignment encoder (LAE). Sigma-conditioned encoders and decoders map both models’ latents into the shared space. Training combines reconstruction, cross-reconstruction, latent alignment, and denoising-velocity alignment losses.
Applications. Once the models share a latent space, a trajectory can be handed off from one model to another mid-denoising. This enables:
Acceleration: hand off between models with different step budgets; Multi-preference composition: combine models optimized for different preferences; Cross-model OPD: on-policy distillation of model B’s velocity field v from model A in the shared latent space
This repository includes LAE training, OPD training, and LAE-based model handoff inference.
Visual results
LAE reconstruction quality
The examples below compare the baseline with the LAE reconstruction results for Qwen-Image, FLUX.1, and SD3. The figure also reports mean SSIM over 100 images for each model.
These examples show the results of switching models and continuing denoising with the second model. The figure includes model pairings beyond the SD3/Qwen-Image inference implementation currently available in this repository.
Quick start
Trained InstantFusion checkpoints are available on Hugging Face.
1. Set up the environment
Use a compatible DiffSynth-Studio checkout in the same environment. The training scripts can import it through DIFFSYNTH_ROOT:
Provide a Qwen-Image model directory containing transformer/, text_encoder/, vae/, and tokenizer/, plus an SD3 single-file checkpoint. Model weights are not bundled with this repository.
2. Prepare the dataset
We use VisPrompt5M for training, which is not included in this repository.
The training data is supplied locally and is not included in this repository:
metadata.csv needs file_name and text columns. Image paths in file_name are relative to DATASET_ROOT/train/:
3. Train the latent alignment encoder
--prepare-prompt-cache creates the required SD3 and Qwen-Image prompt embeddings before training. Training uses ./prompt_cache unless --prompt-cache is set. The cache is runtime data and should stay outside version control.
The LAE checkpoint stores the shared-latent bridge parameters.
4. Train cross-model OPD
Use the LAE checkpoint from the previous stage. OPD trains an SD3 LoRA with Qwen-Image as the frozen teacher. By default it uses 20 sampling steps and supervises zero-based steps 1,2,4,8. Change these with --sampling-steps and --supervised-step-ids when needed.
OPD reuses the LAE prompt cache by default. If you use a different dataset for OPD, add --prepare-prompt-cache to populate embeddings for its prompts.
5. Run LAE-based inference
Advanced settings
The command-line interfaces expose paths and common controls. Put less frequently changed trainer or model settings in a JSON file and pass --config path/to/config.json to the relevant command above. JSON uses the Python setting names with underscores; model and dataset paths remain command-line arguments. For example, an OPD config can contain:
To reuse the training prompt cache during inference, an inference config can contain:
Explicit command-line options take precedence over JSON settings. Training commands also accept the previous underscore-style option names, though --help displays the shorter hyphen-style names. Run bash scripts/train_lae.sh --help, bash scripts/train_opd.sh --help, or bash scripts/infer.sh --help for the compact option lists.
Repository layout
Citation
The paper link and BibTeX entry will be added when the paper is available.
An official implementation of "InstantFusion: A Shared Latent Interface for Heterogeneous Image Generators"