German team proposes RUMI robot teaching system, integrating AR real-time feedback and haptic interaction
A joint team from Technische Universität Nürnberg, the Technical University of Munich, and Siemens has proposed the RUMI robot teaching system, which integrates VR headset tracking, AR real-time feedback, and haptic interaction, and adopts a handheld interface compatible with the Robotiq 2F-85 gripper and replaceable fingertip sensors. Experiments show that after adding real-time robot visualization and pose initialization, the number of rounds with joint limit violations across 50 operations dropped from 16 to 3.
Body · AI translation
English text is being prepared.
Recently, a joint research team from Nuremberg University of Technology, the Technical University of Munich, and Siemens published a paper on the arXiv platform titled "RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback," proposing RUMI, a robot manipulation data collection system that integrates VR headset tracking, AR real-time feedback, and haptic interaction, aiming to improve the feasibility of motions during robot teaching and to support force data collection.
It is understood that when robots perform operations such as grasping and carrying, they need to precisely control the force of contact with objects, but large-scale robot training data with force information is still relatively limited. Although existing handheld teaching devices can collect force information during manipulation, most are designed for simple linear grippers and are difficult to match the complex motion structure of actual robot grippers.
RUMI adopts a handheld manipulation interface matched to the Robotiq 2F-85 robot gripper and is equipped with replaceable fingertip sensors. This design can preserve the motion geometry characteristics of the target gripper while supporting natural manipulation input before contact and force measurement after contact.
The system integrates tracking functionality into the VR headset, displaying the pose and motion state of the target robot in real time via AR, helping operators judge whether the current action conforms to the robot's kinematic constraints. In addition, RUMI introduces a haptic feedback mechanism, supporting teleoperation with force information in both simulated and real robot environments.
Experimental results show that after adding real-time robot visualization and pose initialization functions, the number of episodes with joint limit violations in 50 operations decreased from 16 to 3. The research team also used two different fingertip sensing methods to collect teaching data for training robot imitation learning policies, and evaluated the ability of the haptic interface to collect force data in a simulated environment.
This research demonstrates another application of XR devices in robot learning, namely assisting robot teaching through AR visualization and haptic interaction, providing a new technical solution for embodied intelligence training data collection.
Source: arXiv
Submission/News Tips: tougao@youxituoluo.com
Article/Business Cooperation: Lin Nan (WeChat 19250561593)
Join the industry exchange group: Lin Nan (WeChat 19250561593)
English text is being prepared.
Digital Technology Industry Information and Service Platform
English text is being prepared.