Thread by @wintonARK

some thoughts on tesla AI day



starting with Tesla Vision

🧵

The Tesla Vision work is extraordinary.

And vision is the right word for it.

(Massive kudos to @karpathy and team.)

/1

The black magic of the system: its ability to iteratively predict future driving surfaces, intersection shapes, exhibit object persistence, and generally provide an intuited truth-state of the world onto which prediction, policy and planning can be built.
/2

That Tesla has managed to execute this vision system on end-point devices an ~order of magnitude less expensive than competing autonomous systems is both testament to the brutally parsimonious approach to the underlying problem and indicative of Tesla's strategic advantage.
/3

The hydranet architecture, which provides a pretrained trunk out of which split end-feature "heads" solves the problem of running the net latency-efficient on the end-device hardware and allowing disparate teams to iterate asyncronously on their particular problem-area.

/4

Periodically the entire system including the trunk gets re-trained and (I think) improvements in any individual head (say traffic light detection), will also have the advantage of improving the performance of the trunk (and so improve other outputs as well.)

/5

Of course a nice deep neural architecture is useless without the data to feed it. Since at least the introduction of its first autopilot board Tesla has had data option-value that outpaces its competitors. A frequent counterargument has been "well most of that data is boring"

/6

2 things are apparent (in this presentation and prior)

1: Tesla exploits the option value of its data resource to extract not-boring data for the particular corner-case they are trying to solve.

/7

(In this presentation the example used was vision-only depth estimation of a vehicle in front with debris obscuring view; with a neural net the task becomes trivial when you can scrape 10k truth-labeled examples of a similar situation in a few weeks.)



/8

2: Boring data is not as boring as you might think,

(IFF the data can be very inexpensively labeled and ingested, and if the root training task is to perceive and forecast the road surface and the dynamic and static objects using and surrounding it. )

/9

The amount of work that has gone into de-frictioning the labeling process

-Moving from 3rd party, to in-house;
-2d to 3d/4d;
-Hand-labeled to massively auto-labeled

has produced remarkable results.

/10

Using much more computationally intense neural nets to create the autolabeling tools that then clean the data to train the parsimonious net that can operate on the car-computer

is building a data machine that builds a data machine that runs the data machines

/11

Inexpensive marginal labeling capability matched against a massive queryable dataset allows Tesla to build a generalized model of the road-world (extensible to all geographies that the company has penetrated).

/12

As described in the presentation, the system presents spookily-good predictions of the road and objects ahead (early in the FSD beta release some accused the company of using HD-mapping b/c the vehicles assessment of the surroundings seemed so preternatural.)

/13

This is both the framework on which a generalized driving system can be built, and the way in which they built it provides the likely direction of travel for the other components in the system as they advance.

(More on this later when I have time).

/14