1) Tesla's Autopilot driver assist product is now largely feature complete, but how much further does it have to progress to achieve reliability 3-5x greater than the average human & allow removal of human supervision?
2) This is the March of 9s.
Tesla’s self driving strategy from the start was chosen & optimised for this moment now; putting a system & infrastructure in place without data or hardware bottlenecks to allow largely automated progress for a feature complete AP on the March of 9s.
3) But how far do they have left to March?
Is Tesla 99.9% there, does it have 9,000x further to go or its it 33% there? It’s all a matter of perspective.
And most importantly does this all mean is Tesla 1 year away or 10 years away from Robotaxi level reliability?
4) I’ll get into the details soon, but to skip to the conclusion – I’m leaning towards 33% of the way there and 2 years away from 3x human reliability, with 2021-2024 the most likely 3 year period at ~40% odds.
5) On a high level first, Tesla’s solution to the March of 9s is “Operation Vacation”; aside from efficient human labeling & algorithmic & driving policy tweaks it is largely automated:
A) Fleet disengagements/shadow mode disagreements generate a priority edge case class list
6) B) Fleet collects data when cars see these cases (~1.2bn fleet miles/month)
C) New data is labelled & used to retrain neural nets. System iteratively improves its predictions using self supervised learning using real knowledge of object's future locations.
7) D) System tests the new neural nets in simulation against test cases & chooses the best NN.
E) NN & broader architecture is rolled out in shadow mode and beta to test real world performance.
8) Now to the numbers:
First lets define reliability as probability of zero accidents per 5 seconds of driving (5 seconds chosen somewhat arbitrarily as this is likely the video length used for training Tesla’s NNs).
A) I’ll assume a self driving car is competing against ~9% odds (no controls, just luck) –1 nine.
B) Humans crash in the US every ~100k miles (only serious enough to be reported to NHTSA every ~265k miles, which with~1.8x cars per crash leads to their 1 crash/478k miles data).
10) Translated into probability of no crash per 5 seconds (using 31mph average speed) 100k miles is 99.99996% or 6.6 nines. So to get to 3-5x humans would require ~7 nines.
11) C) On rough estimate I guess Tesla FSD Beta has a disengagement every ~3 miles in the city & Autopilot every ~80 miles on highways. Time & speed weighted this puts Tesla miles per disengagement at 34 & probability of no disengagement per 5 seconds at 99.87%.
12) So Tesla is 99.9% there? Or does it have to reduce disengagement probability per 5 seconds another ~8900x from 0.13% to 0.000014% to get to 3x humans?
I don’t think either of these are the right way to look at it.
13) This is how I think about it:
For Tesla to solve a another 9 it has to get from 99.9% to 99.99% - this requires solving new edge case classes that add up to 0.09% disengagements per 5 seconds.
14) Say solving this 9 requires solving 100 new edge cases. Then it has to solve 100 edge cases which on average individually lead to 0.09%/100 = 0.0009% disengagements per 5 seconds.
15) So they solve these, now they are at 99.99%. To solve the next 9 they need to solve edge cases which add up to 0.009%. They can do this by solving another 100 edge cases which this time on average lead to 0.00009% problems per 5s.
16) This is key; the latest 9 took the same amount of work on labeling & Autopilot team problem solving as the prior 9 – it didn’t need 10x more work. What it did require though is access to 10x more data & data filtering processing (but 1x not 10x more actual data collection).
17) This is where Tesla’s fleet vision strategy shines. It already has a fleet driving ~1.2bn miles/month with HW2+ supercomputers able to do data filtering & processing. So to access 10x more data Tesla simply has to scale up utilisation rates of its customer car fleet.
18) So for example above for getting to 99.999% level (or 2 nines past current FSD Beta), it needed to solve 100 new edge case classes averaging 0.00009% occurrence per 5s. These would happen 25,000 per month across Tesla’s fleet’s 1.2bn miles (38mn hours).
19) So if Tesla needs e.g. 1000 examples of each of these for the NN training solution, it still only has to collect data from 1000/25000 = 4% utilisation of fleet encounters with these cases.
20) So in summary I’d say Tesla has solved two 9s with FSD beta & has to solve four more to get to 3x humans. Hence ~33% there. But when was the starting point? I think @karpathy started working on video based Neural nets in 2018.
21) But probably this latest architecture has only been broadly settled on for ~1 year. Hence Tesla's March of 9s strategy for FSD Beta has progressed 33% in 1 year & thus could get to 3x human in another 2 years.
22) But Tesla is venturing very far into the unknown here. We do not know if they will hit other architecture or hardware bottlenecks or local maximum to slow down their progress. Fortunately Tesla’s agility will help respond to challenges as they come.
23) Ideally Tesla will achieve 3x with the current HW3 hardware, but next year manufacturing capacity will be ~1 million so even if they have to make changes to computing power, camera definition, laser camera cleaners, radar etc, they will very rapidly have a large fleet anyway.
24) Some important notes on this thread: Tesla is solving edge case classes & not specific edge cases; the system has some level of generalization ability.
25) So many very rare edge cases (with a frequency level associated with a later nine) can already be handled by Tesla’s cars by generalising solutions across its playbook of already trained &solved edge case classes.
26) And the larger & more varied the number of edge case classes solved in Tesla’s NNs, the better & better it will get at generalising to never before seen situations.
27) Another important note: All the numbers above are just examples & numbers are just for illustration. The real situation is much more complicated.
28) In particular we do not know the real frequency distribution of edge case classes (particularly the ones which aren’t already solved by generalisation between past class solutions).
29) It could be there is a sudden fatter tail of far rarer new edge case classes needed to solve the next 9; this would then need more actual work & labeling than the past 9 as well as access to even rarer events.
30) Fortunately with Tesla’s fleet driving 13.5bn miles a year it has a lot of fleet driving experience utilisation runway to play with.
31) One argument against a fatter & fatter tail of ungeneralisable edge cases is that if this were true – it seems very unlikely Tesla & Waymo etc would have got as far as they have already.
32) For example if there were say 1 million different 0.0000009% per 5 second edge case classes - this frequency group of problems would add up to 0.9% problems per 5 seconds - & it would have been impossible to solve even the second 9.
33) If I haven’t lost & confused you too much already. Some more threads on Tesla’s Autopilot:

General summary of the rational for Tesla’s Data/Intelligence Heavy Vision strategy:
34) Thread on the detection bottleneck holding back the previous Autopilot architecture & how the new 4D architecture may have solved this:
35) Thread on the human resource bottlenecks to Tesla's Autopilot performance and how they've worked to solve this with Operation Vacation:

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Reflex Research

Reflex Research Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @ReflexFunds

22 Oct
According to usually reliable Tesla hacker @greentheonly Tesla looks set to move to a new next generation radar from Arbe.

Tesla at one point reportedly had its own in-house radar under-development but its not clear if this project is still ongoing.
Read 4 tweets
25 Sep
I may be very wrong, but this looks like a proof of concept of Tesla's proposed new Lithium Clay + Salt + Water method.
1) The method appears to have been discovered by accident by SGS working on the Clayton Valley Lithium Project in Nevada, without them noticing the consequences
2) SGS discovered that for some surface lithium clay samples you could leach lithium in water, but then that this didn't work for other subsurface samples.
3) They decided the reason for this discrepancy must be because surface samples must have had "some component of surface enrichment in the form of water-soluble salts".
4)Then they finished and said ok, water doesn't work, we have to use expensive acids instead.
Read 10 tweets
25 Sep
Tesla likely started the Roadrunner cell project in ~2013 & built general battery cell expertise since foundation in 2003. They have also acquired & hired decades of experience.
The analyst/media takeaway from Battery Day looks that Tesla only started the cell project ~a year ago
Tesla has many huge challenges still to solve (with big delay risks) to ramp Kato road cell production to 10GWh by the 3Q21 target (& full rollout of all the BD innovations isn't targeted until 3Q23) but I see a very large misunderstanding on how far they’ve already come already
Tesla revealed 50+ separate but elegantly entwined cell & raw material R&D projects. They're in varying stages of development & Tesla couldn't explain detail in 90mins (& for competitive reasons) but @elonmusk, Drew, @tbc415 & hundreds more achieved a huge step to ending ICEs
Read 4 tweets
13 Sep
From @KelvinYang7's two independent sources (one of which is a supplier's stock offering filing) linked below, I now see very high probability that Tesla is preparing for a final Model Y production rate of 450k at GF3 (together with Model 3 at 200k).
It will obviously take time to ramp to target capacity which may not be hit until late 2021.
Tesla will also have to ramp demand otherwise it may not pull the trigger to ramp from 2 to 3 shifts.
It also needs to ramp cell supply (fortunately battery day is now very soon!)
Read 5 tweets
25 Aug
I’ve been hoping for Tesla’s Gen1 roadrunner cells to deliver ~30% volumetric energy density increase vs Model 3 at pack level. This could be achieved with ~300-310 wh/kg vs GF1 cells at ~250 wh/kg.
Initially i expect largest progress to be cost, footprint & capex.
I expect Tesla has a pipeline of various technology upgrades in earlier stages of R&D for Roadrunner Gen 2-5 cells etc with a path to 400 Wh/kg.
Longer thread on my expectations for Battery day. https://t.co/YpR34bsMad
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!