Apple unveils humanoid robot control system HALO, achieving a 91% task success rate in unfamiliar environments
Apple's research team published the paper "Perceptive Humanoid Motion from Simulation" on arXiv, unveiling the autonomous humanoid robot control system HALO and validating it on the Unitree G1. The system has three layers: task orchestration, motion generation, and motion tracking. The top layer runs Qwen3.6-35B-A3B locally on a Mac mini mounted on the robot. In 360 single-task tests across 6 real-world environments that had not been pre-scanned, the average success rate was 91%; the average success rate for 3 composite tasks was 70%, and among them, the continuous task of bending down to pass an obstacle, walking toward a target, and pushing a box achieved only 50%.
Why read this
AI score 66
The paper presents a hierarchical control architecture and real-world success rate data, offering insight into the practical limits of VLM-driven robot motion generation.
Topics
Body · AI translation
Add crowdsourcing promotion copy Add crowdsourcing promotion copy end Add App-exclusive traffic guidance
Fast Technology News, November 9: Apple's research team recently publicly released the latest research results on HALO, an autonomous control system for humanoid robots. Through a vision-language model (VLM) and motion generation technology, the system enables robots not only to understand human instructions, but also to adjust their movements according to unfamiliar environments, autonomously completing tasks such as pushing boxes, sitting on chairs, and traversing obstacles.
The research team recently published the paper "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, introducing the design and experimental results of the HALO system, and deployed the system on Unitree Robotics' (688836-CN) G1 humanoid robot for verification.
The research focus is on enabling robots to apply motions learned in simulated environments to real spaces that have not been previously scanned or mapped, reducing the robot's dependence on specific environments.
According to the demonstration video released by the research team, the G1 robot equipped with HALO can, following instructions, push boxes, follow humans, sit down on chairs, and also turn sideways to pass through a half-open door, or bend over and crawl under a table. These actions are not merely pre-set paths; they also involve environment recognition, spatial judgment, and adjusting body posture according to obstacles.
HALO's core design splits the robot's control process into three levels: task orchestration, motion generation, and motion tracking, which respectively handle instruction understanding, whole-body motion planning, and actual joint control.
The top-level task orchestration module uses a vision-language model, with Alibaba's Qwen3.6-35B-A3B preset for use, and runs locally on a Mac mini carried by the robot, processing tasks without an internet connection. The system also supports calling cloud models via API to obtain stronger reasoning and planning capabilities.
This module is responsible for understanding human instructions and, combined with images from the robot's perspective, determining how to complete the task. For example, when receiving the instruction "sit down," the robot must first recognize the surrounding environment and decide on a suitable place to sit; if it receives the instruction "sit on the chair" but the path to the chair is blocked by a table, it must break the task down into a series of continuous actions such as bending over to pass the obstacle, approaching the chair, and sitting down.
During task execution, the module also continuously confirms through video from the robot's perspective whether the action was successful, and then decides whether to proceed to the next step or try again, issuing only one clear subtask to the next level at a time.
The middle-level motion generation module uses another model, Qwen3.5-2B, together with a DiT decoder responsible for generating motions, to plan whole-body motions for approximately the next 5 seconds based on environmental images and the current task. The system replans about 3 times per second, enabling the robot to continuously correct its motions as the environment changes.
The bottom-level motion tracker is trained through procedures such as reinforcement learning and policy distillation, operates at a frequency of 50 times per second, and combines depth images with the robot's own state to convert the motions planned by the upper levels into actual joint control commands.
In actual testing, the research team selected 6 real environments that had not been previously scanned or mapped and conducted 360 single-task tests, with an average success rate of 91%.
However, when tasks involve multiple continuous actions, the system's performance still shows a gap. The research team additionally conducted 10 tests each for 3 composite tasks in a single real environment, with an average success rate of 70%.
Among them, the continuous task of "bending over to pass an obstacle, walking toward the target, and pushing the box" had a success rate of only 50%, indicating that the robot still faces challenges when handling multi-step actions.
To train the robot to adapt to different environments, the research team established a large-scale synthetic data generation pipeline, producing approximately 300 hours of primary training data in 1,000 procedurally generated simulated home environments, pairing the images the robot sees, the instructions it receives, and the corresponding motions.
The data generation pipeline is divided into several stages: first, generate motion sequences according to text instructions; then place the motions into virtual rooms; finally, render the images the robot sees while executing tasks through simulation tools.
Taking "sit on the armchair" as an example, the system first breaks the task down into motion segments such as walking forward, turning, and sitting down; then the motion generation model generates a motion sequence based on text prompts, waypoints, and key poses, and filters the results through geometric checks to reduce unreasonable motions such as foot sliding or the body penetrating the ground.
Next, the research team places the motions in different virtual rooms, checks the positional relationship between the robot and the furniture, excludes configurations that may cause collisions, and then renders images from the robot's perspective. The training process also adjusts lighting, color tone, and image blurriness to increase the diversity of visual data, so that the same pipeline can produce complete training data of instructions, images, and whole-body motions.
In terms of computing resources, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to complete the synthesis of the primary training data; the motion generator was trained using imitation learning with 64 B200 GPUs; the motion tracker was trained through reinforcement learning, policy distillation, and other methods using 4 H100 GPUs.
However, HALO has not yet completely eliminated human involvement. The research team pointed out that when adding new physical tasks, personnel still need to write prompts and design key frames for motions, and some motions also require the use of human videos or motion capture materials.
In addition, at the current stage the system mainly relies on visual perception and still finds it difficult to directly grasp contact states. When objects leave the camera's field of view or are occluded, the execution performance of tasks involving more physical contact will decline.
The research team lists tactile perception as a direction for subsequent improvement, hoping to further enhance the robot's motion control capabilities in complex environments.
[This article ends] If you need to reprint, please be sure to indicate the source: Fast Technology
Editor in charge: Deer Antler article_adlist[
Report article content ]article_adlist Statement: This is an exclusive article from Sina. Reproduction without authorization is prohibited. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin
Non-targeted 300*250 button end