We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-world generalization.

We took our robot into homes that were not in the training data and asked it to clean kitchens and bedrooms. More below⤵️ π-0.5 performs hierarchical inference, inferring high-level semantic subtasks ("pick up the plate") followed by actions. It uses a co-training recipe with data from other robots, high-level commands, verbal instructions, and multimodal data from the web.

There are great tokenizers for text and images, but existing action tokenizers don’t work well for dexterous, high-frequency control. We’re excited to release (and open-source) FAST, an efficient tokenizer for robot actions.

With FAST, we can train dexterous generalist policies via simple next token prediction, and get a 5x training speed-up over prior state of the art! Our FAST tokenizer uses the same techniques as JPEG compression to create compressed action tokens, which enable us to solve complicated tasks that could previously only be tackled with diffusion, like folding laundry, cleaning tables etc.

Blog (and paper + code): pi.website/research/fast