蚂蚁 inclusionAI 开源 LLaDA-UI 扩散式 GUI 智能体

inclusionAI; LLaDA-UI

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork1 (1); Star15 (15)You must be signed in to star a repository

Repository files navigation

README

LLaDA-UI is an MoE-based, block-wise diffusion vision-language GUI agent. It combines a native dynamic-resolution vision encoder with the LLaDA2.0-mini-base diffusion language backbone, and generates reasoning and GUI actions through the same block-wise diffusion decoder.

Figure 1. LLaDA-UI GUI-agent performance and qualitative block-wise diffusion decoding. The radar chart and GUI observation remain static while the model output is progressively denoised.

Highlights

Diffusion-native multimodal agent: LLaDA-UI extends a masked diffusion language model to visual understanding and executable GUI interaction; MoE architecture: approximately 16.7B total parameters end to end; Two-stage training: large-scale multimodal pre-training aligns a native-resolution SigLIP-initialized ViT with LLaDA2.0-mini-base, followed by GUI-agent supervised fine-tuning; Cross-platform interaction: training covers grounding, mobile, desktop, and web tasks, including data from more than 100 Chinese mobile applications and more than 70 English mobile applications; Competitive GUI performance: LLaDA-UI outperforms Qwen2.5-VL-7B across all six reported GUI benchmarks and surpasses Qwen3-VL-8B on ScreenSpot-Pro, AndroidWorld, MobileWorld, and WebVoyager; Diffusion-specific analysis: the accompanying report studies structured action validity, trajectory-length effects, EOS handling, denoising steps, and block size

Model Overview

Training and Inference Pipelines

Inference Pipeline

Figure 2. GUI observations from web, mobile, and desktop environments are encoded at native resolution and combined with task and interaction-history tokens. The LLaDA2.0 decoder progressively denoises the model output into reasoning and executable actions.

GUI Data Generation

Figure 3. Overview of the GUI data-generation pipeline, from task construction and sub-skill decomposition to compositional trajectory collection and quality control.

Benchmark Results

GUI-Agent Evaluation

GUI-agent evaluation reproduced directly from Table 3 of the technical report.

Evaluation versions, task counts, prompts, reset policies, action limits, serving backends, and baseline provenance will be documented in the final release.

Qualitative GUI-Agent Traces

The following GIFs replay complete successful trajectories. Every frame shows the original screenshot together with the verbatim model output for that step.

WebVoyager: constrained flight search

LLaDA-UI configures a one-way Calgary-New York flight, navigates to December 3, 2026, applies the lower-emissions constraint, and returns the verified lowest-CO2 itinerary over a 20-step trajectory.

OSWorld: Calc to Writer

LLaDA-UI selects a formatted table in LibreOffice Calc, transfers it to Writer, and saves the resulting document as price.docx on the desktop.

MobileWorld: email to alarm

LLaDA-UI reads the 7:00 PM Christmas-party time from email, computes the one-hour offset, opens the Clock application, and verifies the enabled 6:00 PM alarm.

Quickstart

Use separate environments for standalone Hugging Face inference and SGLang serving because their tested Transformers and PyTorch versions differ.

Hugging Face inference

inference/inference_hf.py is a self-contained grounding entry point; it does not require the training repository.

Download the checkpoint from the LLaDA-UI Hugging Face repository, then pass its local directory to --ckpt:

The script prints the verbatim generation and parsed normalized point. Use IMAGE_MAX_PIXELS=1003520 for the default-resolution profile. ScreenSpot-V2 pipeline verification is also available; see python inference/inference_hf.py --help.

SGLang serving

inference/sglang_client.py is a single-file, standard-library client for an OpenAI-compatible endpoint. Use a dedicated serving environment and follow the minimal server recipe:

Then replay the packaged mobile, desktop, and web requests:

Each JSON embeds its screenshot and an evaluated current-image-only multi-turn request. The packaged step-$t>0$ cases use the exact role sequence system, user(""), assistant(previous), user(current): the empty historical user turn is followed by the latest raw ... ... response, while the final user turn contains text first and exactly one image, the current screenshot. Historical screenshots are never replayed. Mobile keeps its task/history headings in the current user caption; desktop and web keep the task in the system prompt and use Current Screenshot: as the current caption. At step 0, omit the empty user and previous assistant messages. Send another compatible request with python3 inference/sglang_client.py --request request.json.

GUI Action Format

For navigation, the model maps the task, current screenshot, and available interaction history to tagged reasoning and a text-serialized action. A typical response has the following form:

Coordinates are integer-normalized to [0, 999]. The action vocabulary is platform aware:

Mobile: Click, DoubleClick, LongPress, Drag, Swipe, Type, LaunchApp, Wait, CallUser, GetScreenshot, device navigation, Answer, and Finished; Desktop: pointing actions, RightClick, Drag, Swipe, Type, Hotkey, Wait, CallUser, and Finished; Web: pointing actions, Drag, directional Scroll, Hover, Type, URL Launch, Hotkey, browser navigation, CallUser, and Finished; Grounding: directly returns [x,y], with [-1,-1] for an infeasible request, rather than returning a navigation action

Following the dominant training-data format, point-based navigation actions use box=(x,y). Here box denotes one normalized interaction coordinate rather than a rectangular region.