Thread by @Tesla_AI

Tesla is building the foundation models for autonomous robots twitter.com/i/web/status/1…

Our multi-modal neural networks are already in customer vehicles—these networks take in arbitrary modalities such as camera videos, maps, navigation, IMU (Inertial Measurement Unit), GPS etc.

Tasks such as Occupancy prediction are already quite general in what they represent—in some ways, they are ontology-free & simply predict the probability that some 3D position is occupied.

Such occupancy can be used for collision avoidance by any robot.

All of this is enabled by fleet scale auto-labelling. By using video data from multiple trips in the same location, we can reconstruct the entire scene

In addition, we’re building off state-of-the-art generative modeling techniques—enabling us to predict possible outcomes given past observations, in a jointly consistent manner across multiple camera views

These imagined futures can be action-conditioned to produce different outcomes.

For example, the videos below are generated entirely by the neural network by simply using different prompts

These models will learn from a huge set of extremely diverse data from the Tesla fleet

And will be trained on enormous amounts of compute

These video foundation models will serve as the brain of both the car & Optimus robot 🚘🧠🤖

Join the Tesla AI team to build the future of robotics!

→ tesla.com/AI twitter.com/i/web/status/1…