Thread by @Christiano92

1/ Setting the record straight: What you see here is Sim 1.0, while the industry is moving towards Sim 2.0. Example are @Waymo Simulation City and @RaquelUrtasun GeoSim. Ex: the below reconstruction was manually recreated in Unreal Engine 4 and would take *hours/days. #TeslaAIDay

https://twitter.com/WholeMarsBlog/status/1429537335599792128

2/ Even environments recreated with procedural tools (1.0 tech) are limited by hand modeled assets & textures by artists, introducing huge domain gaps. Its not scalable & is why Sim 1.0 is being depreciated (taking backseat) in some AV companies while Tesla is just introducing it

3/ .@theinformation said in Q4 2018 that Tesla's simulation were "in their infancy". Sim 1.0 is ~2015 era tech that gives perfect labeled ground truth, procedural scenario generation & reconstruction, etc. @aurora_inno goes into details of Sim 1.0 here.
aurora.tech/blog/scaling-s…

4/ Sim 2.0 intends to bridge the domain gap & generate realistic simulation of every part of the AV stack at scale. So instead of using a game engine to create an environment, its automatically generated from real world data with dynamic objects removed/added using NNs.

5/ That same environment is then populated by realistic sensor sim of objects/actors. Ex: GeoSim automatically generates synthetic and realistic geometry & traffic aware camera data for training & testing at scale. Unlike Tesla, manually designing 3d models in 3d max/maya/blender

6/ The 3d models are extracted from real camera data and then shaded, realistically lit and shadowed using neural rendering. Each picture below contains a GeoSim vehicle. Can you tell me what the simulated cars are?

7/ Instead of doing manual labor and hiring 3D environmental artists to model 3d objects and environments. Then spend compute resources to render them. You automatically generate it from video footage in an instant. Each example posted below again has sim cars, which ones is it?

8/ In Tesla's scenario. Geosim & other NN technique that @RaquelUrtasun team have developed would give you the ability to remove & add car/peds in the video. Have 10 real peds crossing the street instead of one. Generate thousands of new scenarios using the same cars in the video

8/ Change their speed, position, trajectory & behavior (aggressive, cautious, drunk). Sim 2.0 involves the realistic generation of sensor data. Camera/lidar/radar, not just from a game engine like Sim 1.0 but from a NN at SCALE while bridging the domain gap. @Waymo's sim lidar

9/ Raquel Urtasun goes in-depth on Sim 2.0 here. While @Tesla is just starting work on "neural rendering techniques" others are deploying GeoSim & Simulation City in their production pipeline that renders sim 1.0 obsolete.

10/ At least Tesla has now changed its mind on simulation from "its grading your own homework" to now "we are excited about what sim can achieve". Here you see SurfelGAN which Waymo's Simulation City is based on.

11/ One thing @Tesla lack when discussing their simulation is disclosing how they are working to incorporate complex smart agents that represent the entire distribution of human driving behavior. Drago from @Waymo goes in-depth on it here. 42 mins