LeRobot 开源真实机器人 AI 工具链，含 SO-100 教程与预训练策略

Build Your Own SO-100 Robot!

Meet the SO-100 – Just $110 per arm! Train it in minutes with a few simple moves on your laptop. Then sit back and watch your creation act autonomously! 🤯

Get the full SO-100 tutorial here.

Want to take it to the next level? Make your SO-100 mobile by building LeKiwi! Check out the LeKiwi tutorial and bring your robot to life on wheels.

LeRobot: State-of-the-art AI for real-world robotics 🤗 LeRobot aims to provide models, datasets, and tools for real-world robotics in PyTorch. The goal is to lower the barrier to entry to robotics so that everyone can contribute and benefit from sharing datasets and pretrained models.

🤗 LeRobot contains state-of-the-art approaches that have been shown to transfer to the real-world with a focus on imitation learning and reinforcement learning.

🤗 LeRobot hosts pretrained models and datasets on this Hugging Face community page: huggingface.co/lerobot Examples of pretrained models on simulation environments

ACT policy on ALOHA env TDMPC policy on SimXArm env Diffusion policy on PushT env

Acknowledgment Thanks to Tony Zhao, Zipeng Fu and colleagues for open sourcing ACT policy, ALOHA environments and datasets. Ours are adapted from ALOHA and Mobile ALOHA. Thanks to Cheng Chi, Zhenjia Xu and colleagues for open sourcing Diffusion policy, Pusht environment and datasets, as well as UMI datasets. Ours are adapted from Diffusion Policy and UMI Gripper. Thanks to Nicklas Hansen, Yunhai Feng and colleagues for open sourcing TDMPC policy, Simxarm environments and datasets. Ours are adapted from TDMPC and FOWM. Thanks to Antonio Loquercio and Ashish Kumar for their early support. Thanks to Seungjae (Jay) Lee, Mahi Shafiullah and colleagues for open sourcing VQ-BeT policy and helping us adapt the codebase to our repository. The policy is adapted from VQ-BeT repo. Installation

Download our source code:

Create a virtual environment with Python 3.10 and activate it, e.g. with `miniconda`:

Install 🤗 LeRobot: **NOTE:** Depending on your platform, If you encounter any build errors during this step you may need to install `cmake` and `build-essential` for building some of our dependencies. On linux: `sudo apt-get install cmake build-essential`

For simulations, 🤗 LeRobot comes with gymnasium environments that can be installed as extras: aloha xarm pusht

For instance, to install 🤗 LeRobot with aloha and pusht, use:

To use Weights and Biases for experiment tracking, log in with

(note: you will also need to enable WandB in the configuration. See below.) Walkthrough

Visualize datasets

Check out example 1 that illustrates how to use our dataset class which automatically downloads data from the Hugging Face hub.

You can also locally visualize episodes from a dataset on the hub by executing our script from the command line:

or from a dataset in a local folder with the `root` option and the `--local-files-only` (in the following case the dataset will be searched for in `./my_local_data_dir/lerobot/pusht`)

It will open `rerun.io` and display the camera streams, robot states and actions, like this:

Our script can also visualize datasets stored on a distant server. See `python lerobot/scripts/visualize_dataset.py --help` for more instructions. The `LeRobotDataset` format

A dataset in `LeRobotDataset` format is very simple to use. It can be loaded from a repository on the Hugging Face hub or a local folder simply with e.g. `dataset = LeRobotDataset("lerobot/aloha_static_coffee")` and can be indexed into like any Hugging Face and PyTorch dataset. For instance `dataset[0]` will retrieve a single temporal frame from the dataset containing observation(s) and an action as PyTorch tensors ready to be fed to a model.

A specificity of `LeRobotDataset` is that, rather than retrieving a single frame by its index, we can retrieve several frames based on their temporal relationship with the indexed frame, by setting `delta_timestamps` to a list of relative times with respect to the indexed frame. For example, with `delta_timestamps = {"observation.image": [-1, -0.5, -0.2, 0]}` one can retrieve, for a given index, 4 frames: 3 "previous" frames 1 second, 0.5 seconds, and 0.2 seconds before the indexed frame, and the indexed frame itself (corresponding to the 0 entry). See example 1_load_lerobot_dataset.py for more details on `delta_timestamps`.

Here are the important details and internal structure organization of a typical `LeRobotDataset` instantiated with `dataset = LeRobotDataset("lerobot/aloha_static_coffee")`. The exact features will change from dataset to dataset but not the main aspects:

A `LeRobotDataset` is serialised using several widespread file formats for each of its parts, namely: hf_dataset stored using Hugging Face datasets library serialization to parquet videos are stored in mp4 format to save space metadata are stored in plain json/jsonl files

Dataset can be uploaded/downloaded from the HuggingFace hub seamlessly. To work on a local dataset, you can specify its location with the `root` argument if it's not in the default `~/.cache/huggingface/lerobot` location. Evaluate a pretrained policy

Check out example 2 that illustrates how to download a pretrained policy from Hugging Face hub, and run an evaluation on its corresponding environment.

We also provide a more capable script to parallelize the evaluation over multiple environments during the same rollout. Here is an example with a pretrained model hosted on lerobot/diffusion_pusht:

Note: After training your own policy, you can re-evaluate the checkpoints with:

See `python lerobot/scripts/eval.py --help` for more instructions. Train your own policy

Check out example 3 that illustrate how to train a model using our core library in python, and example 4 that shows how to use our training script from command line.

To use wandb for logging training and evaluation curves, make sure you've run `wandb login` as a one-time setup step. Then, when running the training command above, enable WandB in the configuration by adding `--wandb.enable=true`.

A link to the wandb logs for the run will also show up in yellow in your terminal. Here is an example of what they look like in your browser. Please also check here for the explanation of some commonly used metrics in logs.

Note: For efficiency, during training every checkpoint is evaluated on a low number of episodes. You may use `--eval.n_episodes=500` to evaluate on more episodes than the default. Or, after training, you may want to re-evaluate your best checkpoints on more episodes or change the evaluation settings. See `python lerobot/scripts/eval.py --help` for more instructions. Reproduce state-of-the-art (SOTA)

We provide some pretrained policies on our hub page that can achieve state-of-the-art performances. You can reproduce their training by loading the config from their run. Simply running:

reproduces SOTA results for Diffusion Policy on the PushT task. Contribute

If you would like to contribute to 🤗 LeRobot, please check out our contribution guide.

Add a pretrained policy

Once you have trained a policy you may upload it to the Hugging Face hub using a hub id that looks like `${hf_user}/${repo_name}` (e.g. lerobot/diffusion_pusht).

You first need to find the checkpoint folder located inside your experiment directory (e.g. `outputs/train/2024-05-05/20-21-12_aloha_act_default/checkpoints/002500`). Within that there is a `pretrained_model` directory which should contain: `config.json`: A serialized version of the policy configuration (following the policy's dataclass config). `model.safetensors`: A set of `torch.nn.Module` parameters, saved in Hugging Face Safetensors format. `train_config.json`: A consolidated configuration containing all parameter userd for training. The policy configuration should match `config.json` exactly. Thisis useful for anyone who wants to evaluate your policy or for reproducibility.

To upload these to the hub, run the following:

See eval.py for an example of how other people may use your policy. Improve your code with profiling

An example of a code snippet to profile the evaluation of a policy: Citation

If you want, you can cite this work with:

Additionally, if you are using any of the particular policy architecture, pretrained models, or datasets, it is recommended to cite the original authors of the work as they appear below: Diffusion Policy ACT or ALOHA TDMPC

VQ-BeT Star History