Apple research team discloses humanoid robot system HALO, verified on Unitree G1 with an average success rate of 91%
Apple's research team submitted the paper "Perceptive Humanoid Motion from Simulation" to arXiv, disclosing the autonomous humanoid robot control system HALO and verifying it through deployment on a Unitree G1. HALO has three layers: a task orchestration module based on Qwen3.6-35B-A3B runs locally on a Mac mini carried by the robot, Qwen3.5-2B serves as a frozen encoder to plan full-body motions of about 5 seconds, and a reinforcement-learning-trained motion tracker outputs joint commands 50 times per second. The team completed 360 single-task tests in 6 real environments that were not pre-scanned, achieving an average success rate of 91%, and an average success rate of 70% on 3 combined tasks.
Why read this
AI score 70
The paper discloses HALO's three-layer control architecture and synthetic data pipeline, offering insight into how humanoid robots transfer simulated motions to real rooms.
Topics
Body · AI translation
Related topics top start Related topics top end Futures promotion begin Futures promotion end Miaopai begin Miaopai end Video player start Video player end Market chart begin Market chart end
Apple's research team discloses progress on the robot "brain."
On November 4 local time, Apple's research team submitted the paper "Perceptive Humanoid Motion from Simulation" to the preprint platform arXiv, disclosing the humanoid robot autonomous control system HALO. The research team deployed the system on Unitree's G1 humanoid robot for verification, enabling the robot to carry the motions learned in a simulated environment into unfamiliar real-world rooms for execution.
The research team also released the data generation, model training and deployment code, covering motion generation, motion tracking, task orchestration, and simulation and real robot operation. A Unitree G1 robot equipped with the HALO system demonstrates tasks such as crawling under a table, sitting on a sofa and pushing a box in different real environments. Image source: HALO research team
The project page shows the robot following instructions to push a box, follow a person, sit on a chair, and turn sideways to pass through a half-open door, bend down or crawl under a table, and other motions. These tasks require the robot to understand its surroundings and adjust its body posture according to spatial conditions.
When executing tasks, HALO is controlled cooperatively by three levels. The topmost layer is the task orchestration module based on a vision-language model (VLM), which by default adopts Qwen3.6-35B-A3B, runs locally on a Mac mini carried by the robot, shares the model serving engine with the motion generator, and can work without an internet connection. The system can also optionally call a cloud model via API to obtain stronger reasoning and planning capabilities.
This layer needs to handle ambiguous information in instructions and arrange motions according to the environment. For example, "sit down" does not explicitly specify a target, so the robot needs to judge from the scene in front of it where to sit; "sit on the chair" can be executed directly when the path is clear, but if a table blocks the way, it needs to be broken down into consecutive subtasks such as bending down to pass, approaching the chair, and sitting down. The task orchestration module continuously observes the environment through video from the robot's perspective, judges whether the current motion is successful, and accordingly advances the plan or retries, issuing one clear subtask to the motion generator at a time.
The middle motion generation layer adopts another Qwen model, Qwen3.5-2B. The research team uses it as a parameter-frozen encoder, which works with a DiT decoder responsible for generating motions to plan the full-body motion for about the next 5 seconds based on the images the robot sees and the current subtask. This layer replans about 3 times per second, enabling the motion plan to be continuously updated as the environment changes.
The bottom layer is a motion tracker trained through reinforcement learning, policy distillation and other steps, running at a frequency of 50 times per second, combining depth images and the robot's own state to convert the motion plan into joint control commands.
The three layers respectively undertake task arrangement, motion planning and body control, collaboratively completing the process from understanding instructions to executing motions.
HALO's three-layer control architecture: the vision-language model is responsible for task orchestration, the motion generation module plans full-body motions, and the reinforcement-learning-trained motion tracker executes joint control. Image source: screenshot collage from the HALO project page
According to the paper, the research team conducted 360 single-task tests in 6 real environments that were not pre-scanned or mapped, with an average success rate of 91%. In addition, in one real environment, 10 tests were conducted for each of 3 combined tasks, with an average success rate of 70%. Among them, the combination of "bend down to pass an obstacle—walk toward the target—push the box" had a success rate of 50%.
How the robot obtains the data needed to complete these tasks is a question this research focuses on. The research team synthesized about 300 hours of main training data in 1,000 procedurally generated simulated home environments, pairing what the robot "sees," what instruction it "receives" and "how it moves."
Specifically, the system first generates motions, then places the motions into virtual rooms, and finally generates images from the robot's perspective. The project page takes "sit on the armchair" as an example, breaking it into segments such as walking forward, turning around and sitting down; the motion generation model generates motion sequences based on text prompts, waypoints and key poses, and then filters motions through geometric checks to reduce problems such as foot sliding and the body penetrating the ground.
Subsequently, the system arranges the motions in procedurally generated rooms, checks the positional relationship between the robot and the furniture, seeks placement schemes without collisions, and then uses simulation tools to render the images the robot sees when executing the motions. During training, conditions such as lighting, color tone and image blur are also varied to make the visual input more diverse. In this way, the same pipeline can generate paired data containing instructions, visual images and full-body motions.
HALO's synthetic data generation and training pipeline: first generate motion trajectories, place them into virtual scenes, then render images from the robot's perspective to form paired training data for training the motion generator and motion tracker. Image source: HALO paper
According to the introduction, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to synthesize the main training data. The motion generator was trained through imitation learning using 64 B200s; the motion tracker was trained through reinforcement learning, policy distillation and other steps using 4 H100s.
It should be pointed out that simulation-generated data has not completely eliminated manual participation. The team stated that newly added physical tasks still require manually written prompts and designed motion keyframes, and some motions also used human videos or motion capture footage.
The research team also pointed out that at present the robot mainly relies on visual perception and finds it difficult to directly perceive contact conditions; when objects leave the camera's field of view or are occluded, the performance of tasks involving more contact will decline. Adding tactile perception is one of the directions for subsequent improvement.
The Paper reporter Qin Sheng news_keyword_pub,stock,sz300024&sh688836 iframe below the main text iframe below the main text General statement General statement QR code at the end start QR code at the end start
QR code at the end start QR code at the end start Editor name and work code
Editor name and work code