智源 FlagOpen 开源 Action-Sketcher：用视觉草图连接推理与动作，入选 CVPR 2026
GitHub - FlagOpen/Action-Sketcher: [CVPR 2026] Official implementation of "Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulation" · GitHub
智源 FlagOpen 开源 Action-Sketcher，这是一个面向长程机器人操作、通过视觉草图把推理转化为动作的框架，已被 CVPR 2026 接收。它采用 See-Think-Sketch-Act 循环，由基础模型把高层指令分解为子任务并生成由点、框、箭头组成的视觉草图，再引导底层策略生成动作序列。团队已发布 Action-Sketcher-PI0-LIBERO 模型与推理代码，数据集和训练代码待发布。
正文
当前显示原文，中文全文译文尚未就绪。
FlagOpen; Action-Sketcher
Repository navigation
NotificationsYou must be signed in to change notification settings; Fork20 (20); Star210 (210)You must be signed in to star a repository
Repository files navigation
README; Apache-2.0 license
Make intent visible. Make action reliable.
🔥 Overview
Action-Sketcher operates in a See-Think-Sketch-Act loop, where a foundation model first performs temporal and spatial reasoning to decompose a high-level instruction (e.g., "Clean the objects on the table") into a subtask and a corresponding Visual Sketch. This sketch, composed of primitives like points, boxes, and arrows, serves as an explicit, human-readable plan that guides a low-level policy to generate robust action sequences. This methodology enables three key capabilities: (bottom left) long-horizon planning through task decomposition, (bottom middle) explicit spatial reasoning by grounding instructions in scene geometry, and (bottom right) seamless human-in-the-loop adaptability via direct sketch correction and intent supervision.
2026-02-22: 🔥🔥🔥 Action-Sketcher gets accepted to CVPR 2026! See you in Denver, Colorado, USA!; 2026-02-22: 🤗 We released Action-Sketcher-PI0-LIBERO model and inference codes; 2026-01-05: 🔥 We released our Project Page of Action-Sketcher
🎯 TODO
Release the model checkpoints and inference codes; Release the full dataset and training codes; Release the dataset generation pipeline and GUI tools (Maybe 1 month)
Action-Sketcher uses a 3-stage training pipeline:
Data Preparation
Download the dataset from 🤗 petersonco/actionsketcher_libero:
The dataset includes:
dataset_index.json: Index file containing paths to episode data; compiled_reasoning.json: Visual reasoning annotations; action_stats.json: Action normalization statistics
Stage 2: Reasoning Training
Train the model to generate visual sketches (reasoning-only, no action prediction):
Stage 3: Joint Action-Language Training
Train both reasoning and action prediction jointly:
Action-Only Fine-tuning (Optional)
Fine-tune on action prediction only (freezes reasoning):
Checkpoint Conversion
Training saves checkpoints in DeepSpeed format. Convert to HuggingFace format for inference:
🤖 Method
The Action-Sketcher framework is model-agnostic and can be integrated with any VLA model with an event-driven loop that (i) summarizes the next subtask, (ii) emits a compact Visual Sketch (points, boxes, arrows, relations) that externalizes spatial intent, and (iii) synthesizes an action chunk conditioned on that sketch and the robot state. The explicit intermediate supports targeted supervision, on-the-fly correction, and reliable long-horizon execution within a single-model architecture.
✨ Experiments
📑 Citation
If you find our work helpful, feel free to cite it: