Apple research team discloses humanoid robot system HALO, with an average success rate of 91% on the Unitree G1
Apple's research team submitted the paper "Perceptive Humanoid Motion from Simulation" to arXiv, disclosing the humanoid robot autonomous control system HALO and validating it on the Unitree G1. HALO has three layers: the top layer uses Qwen3.6-35B-A3B to perform task orchestration locally on a Mac mini carried by the robot, the middle layer uses Qwen3.5-2B together with a DiT decoder to plan full-body motions of about 5 seconds, and the bottom-layer motion tracker outputs joint commands 50 times per second. The team completed 360 single-task tests in 6 real environments that had not been pre-scanned, achieving an average success rate of 91%, and an average success rate of 70% on 3 combined tasks.
Why read this
AI score 67
The paper discloses HALO's three-layer control architecture and synthetic data pipeline, from which readers can learn how humanoid robots transfer simulated motions to real rooms.
Topics
Body · AI translation
Download the app
+1 Average success rate 91%! Apple research team discloses progress on robot "brain": calls Qwen models, validated on Unitree G1
The Paper reporter Qin Sheng 2026-11-07 17:12 Source: The Paper ∙ 10% Company > Listen to full text Font size
Apple research team discloses progress on robot "brain" research.
On November 4 local time, the Apple research team submitted a paper titled "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, disclosing the humanoid robot autonomous control system HALO. The research team deployed the system on Unitree Technology's G1 humanoid robot for validation, enabling the robot to carry the motions learned in a simulated environment into unfamiliar real-world rooms for execution.
The research team also released the data generation, model training and deployment code, covering motion generation, motion tracking, task orchestration, and simulation and real robot operation, among other aspects.
The Unitree G1 robot equipped with the HALO system demonstrates tasks such as ducking under a table, sitting on a sofa and pushing a box in different real environments. Image source: HALO research team
The project page shows the robot following instructions to push a box, follow a person, sit on a chair, as well as turning sideways to pass through a half-open door, and bending down or crawling under a table. These tasks require the robot to understand its surrounding environment and adjust its body posture according to spatial conditions.
When executing tasks, HALO is controlled collaboratively by three levels. The topmost layer is the task orchestration module based on a vision-language model (VLM), which by default adopts Qwen3.6-35B-A3B, running locally on the Mac mini carried by the robot, sharing the model service engine with the motion generator, and able to work without an internet connection. The system can also optionally call cloud models via API to obtain stronger reasoning and planning capabilities.
This layer needs to handle ambiguous information in instructions and arrange motions according to the environment. For example, "sit down" does not explicitly specify a target, so the robot needs to judge from the scene in front of it where to sit; "sit on the chair" can be executed directly when the path is clear, but if a table blocks the way, it needs to be broken down into consecutive subtasks such as bending down to pass, approaching the chair, and sitting down. The task orchestration module continuously observes the environment through video from the robot's perspective, judges whether the current motion is successful, and accordingly advances the plan or retries, issuing one clear subtask to the motion generator each time.
The middle motion generation layer adopts another Qwen model, Qwen3.5-2B. The research team uses it as a parameter-frozen encoder, working with a DiT decoder responsible for generating motions, to plan the full-body motions for about the next 5 seconds based on the images the robot sees and the current subtask. This layer replans about 3 times per second, enabling the motion plan to be continuously updated as the environment changes.
The bottom layer is a motion tracker trained through reinforcement learning, policy distillation and other steps, running at a frequency of 50 times per second, combining depth images and the robot's own state to convert the motion plan into joint control commands.
The three layers respectively undertake task arrangement, motion planning and body control, collaboratively completing the process from understanding instructions to executing motions.
HALO's three-layer control architecture: the vision-language model is responsible for task orchestration, the motion generation module plans full-body motions, and the motion tracker trained through reinforcement learning executes joint control. Image source: HALO project page screenshot collage
According to the paper, the research team conducted 360 single-task tests in 6 real environments that were not pre-scanned or mapped, with an average success rate of 91%. In addition, in one real environment, 10 tests were conducted for each of 3 combined tasks, with an average success rate of 70%. Among them, the combination of "bend down to pass an obstacle—walk toward the target—push the box" had a success rate of 50%.
How the robot obtains the data needed to complete these tasks is a question this research focuses on. The research team synthesized about 300 hours of primary training data in 1,000 procedurally generated simulated home environments, pairing what the robot "sees," what instruction it "receives," and "how it moves."
Specifically, the system first generates motions, then places the motions into virtual rooms, and finally generates images from the robot's perspective. The project page takes "sit on the armchair" as an example, breaking it down into segments such as walking forward, turning around and sitting down; the motion generation model generates motion sequences based on text prompts, waypoints and key poses, and then filters motions through geometric checks to reduce problems such as foot sliding and the body penetrating the ground.
Subsequently, the system arranges the motions into procedurally generated rooms, checks the positional relationship between the robot and the furniture, seeks placement schemes without collisions, and then uses simulation tools to render the images the robot sees when executing the motions. During training, conditions such as lighting, color tone and image blurriness are also varied to make the visual input more diverse. In this way, the same pipeline can generate matched data containing instructions, visual images and full-body motions.
HALO's synthetic data generation and training pipeline: first generate motion trajectories, place them into virtual scenes, then render images from the robot's perspective to form matched training data for training the motion generator and motion tracker. Image source: HALO paper
According to the introduction, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to synthesize the primary training data. The motion generator was trained through imitation learning using 64 B200s; the motion tracker was trained through reinforcement learning, policy distillation and other steps using 4 H100s.
It should be pointed out that simulation-generated data has not completely eliminated manual participation. The team stated that newly added physical tasks still require manually writing prompts and designing motion keyframes, and some motions also used human videos or motion capture footage.
The research team also pointed out that at present the robot mainly relies on visual perception and finds it difficult to directly perceive contact conditions; when objects leave the camera's field of view or are occluded, the performance of tasks involving more contact will decline. Adding tactile perception is one of the directions for subsequent improvement. Responsible editor: Li Yuequn The Paper tip-off: 021-962866 The Paper, unauthorized reproduction prohibited +1 Favorite I want to report #HumanoidRobot #Apple #UnitreeTechnology #Qwen #AI View more