Apple research team releases HALO humanoid robot control system, validated on Unitree G1 with an average success rate of 91%
Apple's research team submitted the paper "Perceptive Humanoid Motion from Simulation" to arXiv, disclosing HALO, an autonomous control system for humanoid robots, and deployed it on a Unitree G1 for validation. The system has three layers: a task orchestration module based on Qwen3.6-35B-A3B runs locally on a Mac mini carried by the robot; Qwen3.5-2B serves as a frozen encoder and, together with a DiT decoder, plans full-body motions of about 5 seconds; and a motion tracker trained with reinforcement learning outputs joint commands 50 times per second. The team completed 360 single-task tests in 6 real environments that were not pre-scanned, achieving an average success rate of 91%, and an average success rate of 70% on 3 combined tasks.
Why read this
AI score 67
The paper discloses HALO's three-layer control architecture and synthetic data pipeline, offering insight into how humanoid robots transfer simulated motions to real rooms.
Topics
Body · AI translation
Main body EM_StockImg_Start
EM_StockImg_End
Apple research team discloses progress on robot "brain" research.
On November 4 local time, the Apple research team submitted the paper "Perceptive Humanoid Motion from Simulation" to the preprint platform arXiv, disclosing the humanoid robot autonomous control system HALO. The research team deployed this system on Unitree's G1 humanoid robot for validation, enabling the robot to carry the motions learned in a simulated environment into unfamiliar real-world rooms for execution.
The research team also released the data generation, model training, and deployment code, covering motion generation, motion tracking, task orchestration, and simulation and real-robot operation, among other aspects.
The Unitree G1 robot equipped with the HALO system demonstrates tasks such as crawling under a table, sitting on a sofa, and pushing a box in different real environments. Image source: HALO research team
The project page showcases the robot following instructions to push a box, follow a person, sit on a chair, as well as turning sideways to pass through a half-open door, and bending down or crawling under a table. These tasks require the robot to understand its surrounding environment and adjust its body posture according to spatial conditions.
When executing tasks, HALO is controlled collaboratively by three levels. The topmost layer is the task orchestration module based on a vision-language model (VLM), which by default adopts Qwen3.6-35B-A3B, running locally on a Mac mini carried by the robot and sharing the model serving engine with the motion generator, so it can work without an internet connection. The system can also optionally call cloud models via API to obtain stronger reasoning and planning capabilities.
This layer needs to handle ambiguous information in instructions and arrange actions according to the environment. For example, "sit down" does not explicitly specify a target, so the robot needs to judge from the scene in front of it where to sit; "sit on the chair" can be executed directly when the path is clear, but if a table blocks the way, it needs to be broken down into consecutive subtasks such as bending down to pass, approaching the chair, and sitting down. The task orchestration module continuously observes the environment through video from the robot's perspective, judges whether the current action is successful, and accordingly advances the plan or retries, issuing one clear subtask to the motion generator at a time.
The middle motion generation layer adopts another Qwen model, Qwen3.5-2B. The research team uses it as a parameter-frozen encoder, which works with a DiT decoder responsible for generating motions to plan full-body motions for about the next 5 seconds based on the images the robot sees and the current subtask. This layer replans about 3 times per second, enabling the motion plan to be continuously updated as the environment changes.
The bottom layer is a motion tracker trained through reinforcement learning, policy distillation, and other steps, running at a frequency of 50 times per second, combining depth images and the robot's own state to convert motion plans into joint control commands.
The three layers respectively undertake task arrangement, motion planning, and body control, collaboratively completing the process from understanding instructions to executing actions.
HALO's three-layer control architecture: the vision-language model is responsible for task orchestration, the motion generation module plans full-body motions, and the reinforcement-learning-trained motion tracker executes joint control. Image source: screenshot collage from the HALO project page
According to the paper, the research team conducted 360 single-task tests in 6 real environments that were not pre-scanned or mapped, with an average success rate of 91%. In addition, in one real environment, 10 tests were conducted for each of 3 combined tasks, with an average success rate of 70%. Among them, the combination of "bend down to pass an obstacle—walk toward the target—push the box" had a success rate of 50%.
How the robot obtains the data needed to complete these tasks is a question this research focuses on. The research team synthesized about 300 hours of main training data in 1,000 procedurally generated simulated home environments, pairing "what the robot sees," "what instruction it receives," and "how it moves."
Specifically, the system first generates motions, then places the motions into virtual rooms, and finally generates images from the robot's perspective. The project page takes "sit on the armchair" as an example, breaking it down into segments such as walking forward, turning around, and sitting down; the motion generation model generates motion sequences based on text prompts, waypoints, and key poses, and then filters motions through geometric checks to reduce problems such as foot sliding and the body penetrating the ground.
Subsequently, the system arranges the motions into procedurally generated rooms, checks the positional relationship between the robot and the furniture, seeks collision-free placement schemes, and then uses simulation tools to render the images the robot sees while executing the motions. During training, conditions such as lighting, color tone, and image blur level are also varied to make the visual input more diverse. In this way, the same pipeline can generate matched data containing instructions, visual images, and full-body motions.
HALO's synthetic data generation and training pipeline: first generate motion trajectories, place them into virtual scenes, then render images from the robot's perspective to form matched training data for training the motion generator and motion tracker. Image source: HALO paper
According to the introduction, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to synthesize the main training data. The motion generator was trained through imitation learning using 64 B200s; the motion tracker was trained through reinforcement learning, policy distillation, and other steps using 4 H100s.
It should be noted that simulation-generated data has not completely eliminated manual involvement. The team stated that newly added physical tasks still require manually written prompts and designed motion keyframes, and some motions also used human videos or motion capture footage.
The research team also pointed out that at present the robot mainly relies on visual perception and finds it difficult to directly perceive contact conditions; when objects leave the camera's field of view or are occluded, the performance of tasks involving more contact will decline. Adding tactile perception is one of the directions for future improvement.
(Article source: The Paper) Middle of main body Embedded advertisement