AI News

Apple publishes research results on HALO, an autonomous control system for humanoid robots

Electronic Technology ApplicationUpdated Nov 9 12:07

AI summary

Apple's research team published the paper "Perceptive Humanoid Motion from Simulation" on arXiv, unveiling HALO, an autonomous control system for humanoid robots, and validated it on the Unitree G1 robot. HALO splits the control pipeline into three layers: task orchestration, motion generation, and motion tracking, with the top layer running Qwen3.6-35B-A3B locally on a Mac mini mounted on the robot. The team conducted 360 single-task tests across 6 previously unscanned real-world environments, achieving an average success rate of 91%, but the average success rate for 3 composite tasks was only 70%.

Why read this

AI score 65

The paper presents HALO's three-layer control architecture and real-world success rate data, offering insight into the specific methods and limitations of VLM-driven robot motion generation.

Topics

Tags

Body · AI translation

On November 9, it was reported that Apple's research team recently published the latest research results on HALO, an autonomous control system for humanoid robots. Through a vision-language model (VLM) and motion generation technology, the system enables the robot not only to understand human instructions but also to adjust its movements according to unfamiliar environments, autonomously completing tasks such as pushing boxes, sitting on chairs, and navigating obstacles.

The research team recently published a paper titled "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, introducing the design and experimental results of the HALO system, and deployed the system on Unitree Robotics' (688836-CN) G1 humanoid robot for validation.

The research focuses on enabling the robot to apply motions learned in simulated environments to real spaces that have not been previously scanned or mapped, reducing the robot's dependence on specific environments.

According to demonstration videos released by the research team, the G1 robot equipped with HALO can, following instructions, push boxes, follow humans, sit down on chairs, sidle through half-open doors, and bend over or crawl under tables. These actions are not merely pre-set paths; they also involve environment recognition, spatial judgment, and adjusting body posture according to obstacles.

HALO's core design splits the robot's control pipeline into three levels: task orchestration, motion generation, and motion tracking, which respectively handle instruction understanding, whole-body motion planning, and actual joint control.

The top-level task orchestration module uses a vision-language model, by default Alibaba's Qwen3.6-35B-A3B, and runs locally on a Mac mini carried by the robot, processing tasks without an internet connection. The system also supports calling cloud models via API to obtain stronger reasoning and planning capabilities.

This module is responsible for understanding human instructions and, combined with images from the robot's perspective, determining how to complete tasks. For example, when receiving the instruction "sit down," the robot must first recognize its surroundings and decide on a suitable place to sit; if it receives the instruction "sit on the chair" but the path to the chair is blocked by a table, it must break the task down into a sequence of actions such as bending over to pass the obstacle, approaching the chair, and sitting down.

During task execution, the module also continuously uses video from the robot's perspective to confirm whether actions have succeeded, then decides whether to proceed to the next step or retry, issuing only one clear subtask to the next level at a time.

The middle-level motion generation module uses another model, Qwen3.5-2B, together with a DiT decoder responsible for generating motions, to plan whole-body motions for roughly the next 5 seconds based on environmental images and the current task. The system replans about 3 times per second, allowing the robot to continuously correct its motions as the environment changes.

The bottom-level motion tracker is trained through processes such as reinforcement learning and policy distillation, operates at 50 times per second, and combines depth images with the robot's own state to convert the motions planned by the upper levels into actual joint control commands.

In practical testing, the research team selected 6 real environments that had not been previously scanned or mapped and conducted 360 single-task tests, achieving an average success rate of 91%.

However, when tasks involve multiple consecutive actions, the system's performance still shows a gap. The research team additionally conducted 10 tests each for 3 composite tasks in a single real environment, with an average success rate of 70%.

Among them, the consecutive task of "bending over to pass an obstacle, walking toward the target, and pushing a box" had a success rate of only 50%, indicating that the robot still faces challenges in handling multi-step actions.

To train the robot to adapt to different environments, the research team established a large-scale synthetic data generation pipeline, producing about 300 hours of primary training data in 1,000 procedurally generated simulated home environments, pairing the images the robot sees, the instructions it receives, and the corresponding motions.

The data generation pipeline is divided into several stages: first, motion sequences are generated according to text instructions; then the motions are placed into virtual rooms; finally, simulation tools are used to render the images the robot sees while executing tasks.

Taking "sit on the armchair" as an example, the system first breaks the task down into motion segments such as walking forward, turning around, and sitting down; then the motion generation model generates a motion sequence based on text prompts, waypoints, and key poses, and filters the results through geometric checks to reduce unreasonable motions such as foot sliding or the body penetrating the ground.

Next, the research team places the motions in different virtual rooms, checks the positional relationship between the robot and the furniture, excludes configurations that may cause collisions, and then renders images from the robot's perspective. The training process also adjusts lighting, color tone, and image blurriness to increase the diversity of visual data, enabling the same pipeline to produce complete training data of instructions, images, and whole-body motions.

In terms of computing resources, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to complete the synthesis of the primary training data; the motion generator was trained using imitation learning with 64 B200 GPUs; the motion tracker was trained through reinforcement learning, policy distillation, and other methods using 4 H100 GPUs.

However, HALO has not yet completely eliminated human involvement. The research team noted that when adding new physical tasks, personnel still need to write prompts and design key motion frames, and some motions also require human videos or motion capture footage.

In addition, at the current stage the system mainly relies on visual perception and cannot yet directly grasp contact states. When objects leave the camera's field of view or are occluded, the execution performance of tasks involving more physical contact declines.

The research team lists tactile perception as a direction for future improvement, hoping to further enhance the robot's motion control capabilities in complex environments.

Source

Electronic Technology Application