Xingyuan Intelligence Releases Cross-Embodiment Data Engine Chorus, Untangling the Challenges of the Embodied Data Pipeline

The scale of data keeps expanding, but its effectiveness has not kept pace.

English text is being prepared.

English text is being prepared.

01

Data volume is rising, but "usable data" has not risen along with it. Raw data usually has to go through five stages before it becomes data that a robot can learn from:

First is data filtering, removing low-quality samples such as blurry images, interrupted actions, and abnormal trajectories;

Second is data annotation, adding information such as task, object state, action stage, and language instructions to the operation process so that the model can understand what is happening in the data;

Third is action conversion, mapping human demonstrations or actions generated by one robot into the action space that the target robot can execute;

Fourth is data validation, using rule checks, simulation tests, or real-robot experiments to determine whether the data is reasonable and executable;

Finally, there is feedback iteration, adjusting data filtering, annotation, and conversion strategies based on model training and actual deployment performance. Most parts of this chain can already be automated, but there are many difficulties in actual operation. First, data lacks unified, fine-grained filtering and evaluation standards. From the experience of large language models, if a batch of dirty data is mixed into training, the resulting intelligence degradation is likely to be permanent and will continue to affect the entire model lifecycle. But the industry still lacks a scalable data filtering mechanism. A video with clear images and complete actions does not mean it is useful for model training; a converted trajectory that satisfies kinematic constraints does not mean the robot can successfully complete the task in a real environment. Data suppliers and model companies have different understandings of "high quality," and how to measure the actual training value of a piece of data remains a difficult problem. Second, there is an obvious threshold in converting human actions into robot actions—that is, action retargeting is difficult. Different robots have different joint structures, workspaces, and end effectors, and the same task may require different operation strategies. Especially in first-person human videos, information such as robot joint states and force sensing is often missing, and the system needs to recover spatial relationships from limited visual observations and then generate actions adapted to the target robot. For robotic arms with similar structures, methods such as inverse kinematics can be used to solve the problem; but if the end effectors differ greatly, the problem becomes much more complex. When a human grasps an object with a five-fingered hand, they can adjust the positions of multiple fingers and the way force is applied; when a gripper performs the same task, it uses a completely different approach angle and grasping strategy. The two are difficult to describe with the same set of action trajectories. In addition to the technical challenges of the data pipeline itself, there are also many chaotic practices in the industry. First, the division of labor in data processing is confused; data collection, data processing, and model training are often undertaken by different teams, and the delivery standards and evaluation metrics of each party may not be consistent. Second, many companies rely on open-source pipelines. Many teams use open-source tools to stitch together processing workflows, but lack systematic design for specific models, tasks, and robot embodiments. Industry discussions tend to focus on how many hours were collected and how many trajectories were accumulated, while disclosure is relatively limited on what processing the data underwent, how many samples were ultimately usable, and whether downstream task performance improved. As a result, data scale, usable data scale, and the scale of data that truly produces training gains are easily conflated. Building a complete data pipeline and using training and deployment results to verify the value of each stage is becoming a problem that needs to be solved more than simply expanding collection volume.

02

The lower the collection barrier, the higher the pipeline requirements. There are more and more sources of embodied data, but the pipeline challenges faced by different data types are not the same. Currently, the industry is mainly exploring four categories: real-robot teleoperation, simulation, UMI, and first-person human video. Real-robot data has the best quality and deployment consistency. Data generated by real robots is close to the target robot's control, sensors, and execution environment, making it easy to verify directly. The cost is that it is expensive: robots, operators, sites, and tasks all require investment; second, cross-embodiment remains difficult. Data collected on robot A needs to be used on robot B, and each company handles it differently, resulting in different cross-embodiment capabilities. The advantage of simulation data is that it is very easy to scale up; the disadvantage is the often-discussed Sim2Real gap. For the many simulators centered on robots, the underlying physics engines have not undergone sufficiently large changes for many years and cannot accurately reflect information such as friction, object deformation, and contact positions. This defect is especially obvious in contact-rich tasks that require precise contact control, such as assembly, grasping, and twisting. Low-cost manipulation collection solutions such as UMI record human operations through lighter collection devices, reducing dependence on a complete robot teleoperation system and making each data sample cheaper and easier to collect. But it still requires people to actively perform tasks and use specific equipment, and there are still action mapping and adaptation problems between different UMI collection devices and robots. In essence, it is still incremental data driven by projects, and diversity and the scaling ceiling are constrained by the way collection is organized. First-person human video opens up a larger-scale data source. There are already a large number of manipulation behaviors in people's daily lives, and devices such as head-mounted cameras and smart glasses have the opportunity to record continuously, reducing the cost of specially organized collection. But this route shifts more pressure to the data pipeline. First, videos usually lack joint states and action information that robot systems can directly use, so the system needs to extract hands, objects, and actions from the video and then recover spatial relationships. Second, collection devices vary more widely, with different image quality, viewpoints, frame rates, and sensor configurations, and the pipeline needs to absorb these differences. The broader the sources, the harder it is to unify the quality of raw data. Everyday videos contain both valuable manipulations and a large amount of content unrelated to training goals, requiring more efficient filtering methods. Third, there are differences between human operations and robot embodiments. The person in the video can complete tasks with dexterous hands, while the robot may be equipped with hands or grippers of different structures, and the system needs to convert human actions into actions that the target robot can execute. The advantages and shortcomings of the four routes can be summarized as follows: real-robot data is closer to real execution, simulation is easier to scale up, UMI lowers some collection barriers, and first-person human video has the opportunity to obtain richer and more natural data sources. The basic contradiction is that reducing restrictions on collection devices helps expand data scale but increases the workload of cleaning, spatial reconstruction, and cross-embodiment conversion. In other words, the richer the data sources, the more necessary it is to have data infrastructure that can uniformly handle different devices, different action representations, and different robot embodiments. More importantly, expanding data sources does not equal directly expanding the experience the model can learn from. If the goal is to continuously expand the scale of usable data and prioritize Scaling, first-person human video and UMI are undoubtedly the routes with the most potential; but to truly use them for robot training, the key is to establish a sufficiently reliable data processing pipeline. This is exactly the direction chosen by Xingyuan Intelligence.

03

English text is being prepared.

04

Model vendors must define their own data pipelines. To evaluate whether an embodied data pipeline is well designed, three questions are key: how much effective training data can be converted from raw data, whether this data can truly improve model capabilities, and whether the cost of the entire process can support scaling. The figures Xingyuan provided are: the average usable-data conversion rate has risen to 85%; under the current test configuration, through adaptation to 16 robot embodiments, the processing cost per hour of effective robot trajectories has been reduced to 15 yuan; combined with 3x instruction augmentation, approximately 40x expansion of effective training data can be achieved, with the equivalent cost per unit of training samples at about 5 yuan/hour. In addition, Continuo, previously released by Xingyuan, can be deployed on wearable devices and supports large-scale collection and continuous understanding of human action videos; Duet, planned for release within the year, is a human-machine co-driving system for real deployment, in which the robot executes autonomously and proactively requests takeover when encountering difficulties, one person can supervise multiple robots on demand, and the correction experience generated by takeovers flows back for subsequent training. Continuo, Chorus, and Duet together constitute Xingyuan's learning infrastructure for moving from real-world experience to physical action. It is not hard to see that Xingyuan's data pipeline, model training, and deployment systems are coupled. Xingyuan believes that model vendors must define their own data pipelines: without knowing what inputs and outputs the target robot needs, one does not know what the pipeline should output; handing over the pipeline is equivalent to handing over the right to interpret scaling. Xingyuan's three Chapters are the unfolding of this philosophy: Continuo addresses where data comes from, Chorus addresses how data becomes training signals, and Duet addresses how experience flows back after deployment. Returning to the question at the beginning: when the industry has already accumulated more and more embodied data, why has the data not yet been fully converted into robot capabilities? Xingyuan's solution is to promote embodied data Scaling through two paths: one is to develop the value of data from humans' daily activities, especially the massive amount of human first-person video, and build a cross-embodiment data pipeline with extremely strong data processing capabilities to reduce the cost of experience conversion; the other is to accumulate real data from robot deployment, allowing the model to continuously acquire new learning experience during execution. To make this data truly serve robots of different forms, the underlying cross-embodiment problem also needs to be solved. This constitutes Xingyuan's overall approach from human data collection and cross-embodiment conversion to deployment feedback. But expanding data only solves half the problem. The relationship between data and models is not a simple supply relationship. The data pipeline determines which real-world experiences can enter the model's learning objectives, while training performance and feedback from real deployment in turn determine what kind of data is more valuable and what kind of processing is needed. Only by linking data production and conversion with model training can the growth of data scale possibly promote the orderly improvement of model capabilities. Yang Shiqi told us that this is also why Xingyuan chose to deeply integrate the data pipeline with model R&D. For robot foundation models, Scaling is not just about continuously increasing training data; it is even more about improving the learning value of each piece of data through mutual iteration between data and models. Of course, this is only one route. Real-machine collection, simulation, UMI, and human video each have their advantages, and there is still room for exploration in data representation, conversion methods, and training mechanisms. Embodied intelligence needs Scaling, but Scaling not only requires large-scale data support, but also requires finding ways to make more data truly effective. Xingyuan's exploration shows the industry that some concrete and feasible engineering solutions for data scaling already exist. This may also be the direction worth watching next for embodied data pipelines: how to enable the rich experience already existing in the real world to enter the robot learning process at lower cost and higher efficiency.