Thread by @AjayMandlekar

Tired of collecting demonstrations all day to train your robot?

Introducing MimicGen, an autonomous data generation system for robotics. Using just 200 human demos we generated a large multi-task dataset of 50K demos! #CoRL2023 #NVIDIAResearch 👇



🧵 1/ mimicgen.github.io

First, a human teleoperator provides a small number (~10) of demos. Then, MimicGen automatically generates 1000s of new demonstrations across diverse configurations, objects, and robots.

🧵 2/

MimicGen produces large-scale datasets with minimal human effort. We used MimicGen to autonomously generate over 50,000 demonstrations from less than 200 human demonstrations across 18 tasks, multiple simulators, and the real-world.

🧵 3/

MimicGen can easily produce large datasets for new object configurations, new robots, and new objects, despite source human demos having just a small number of configurations, a single robot arm, or a single object.

🧵 4/

MimicGen works with a wide variety of tasks including contact-rich manipulation requiring millimeter-precision, long-horizon tasks, and mobile manipulation. It is also simulator-agnostic (contact-rich tasks are from IsaacGym and Factory).



🧵 5/

MimicGen can also be used to generate demonstrations for real-world manipulation tasks. As an example, we collected 10 human demos on this coffee preparation task and generated 100 MimicGen demos.

🧵 6/

MimicGen datasets are readily compatible with any offline policy learning algorithm, but even applying simple Behavioral Cloning to MimicGen datasets can produce performant policies across diverse tasks.

🧵 7/

Using MimicGen to generate equal amounts of data as a human operator can result in comparable policy performance (e.g. 200 MimicGen demos from 10 human demos vs. 200 human demos).

This raises important questions about when to request additional data from a human.

🧵 8/

Surprisingly, MimicGen can also produce good policies across different quality source demos. We generated 1000 demos from 10 better quality demos and 10 worse quality demos from the robomimic study and found that policies trained on each dataset achieve comparable results.

🧵 9/

Datasets, simulation environments, and code to reproduce policy learning results are available at . Full compatibility with the robomimic framework.



🧵 10/ github.com/NVlabs/mimicge…

This work was made possible by a wonderful team! @snasiriany @bowenwen_me Iretiayo Akinola, Yashraj Narang, @DrJimFan @yukez Dieter Fox

Paper:
Website:
Code:

🧵 11/arxiv.org/abs/2310.17596
mimicgen.github.io
github.com/NVlabs/mimicge…