Thread by @eliasmrtnz1

🧵Measuring #FSDBeta progress as a community after 1 year:
Many people like to use the metric “miles to disengagement” to measure or compare progress for autonomous vehicles. While this is a possible metric, is it the right one and the only one? I don’t think so…
1/11

Waymo doesn’t think miles to disengagement provides relevant insights into capabilities or comparison of performance to others:

https://twitter.com/Waymo/status/1232758355082375168?s=20&t=aRQoFneOH8JERkaEimBwbA

Their thread goes on to state that the data may fluctuate depending on geography and other factors.
2/11

Cruise provided 4 different reasons for disengagements: medium.com/cruise/the-dis…
1 reason under ‘True AV limitations or errors’ stated about disengagements “are sometimes reportable to the DMV”. This raises a big question: Are companies only reporting SOME disengagements?
3/11

Same article: “Disengagement data isn’t adjusted for driving complexity, weather conditions, or other spatio-temporal variations between miles that affect difficulty”. The operating area & driving complexity is not apples to apples.
4/11

Why does Tesla progress look flat at ~8 miles to disengagement? Some say location, like “works better in California”. Looking at miles to disengagement by State/Province, we see that most states hover around 10. Complexity varies by urban, suburban, & rural or combination
5/11

Some say maybe a specific or new testers are the cause. Looking at miles to disengagement by Tester, we see that most testers hover between 1.5-15. Again, driving complexity varies by many factors including location.
6/11

What about by driving conditions including weather? Looking at miles to disengagement by driving conditions, we see a big spike in May 2022. This is version 10.11 which was the best version with >1k miles & highest overall Miles2DE. Most of the conditions hover between 6-15.
7/11

An example: I drive 23mi round trip to work every day (testing since Oct ’21). On the way to work, at least 2 disengagements that were lane issues happened in exact same spot. On the way home, at least 1 disengagement that was late braking for change in speed limit 45>25mph.
8/11

In Jun ’22, one lane issue was fixed. In Jan ’23, late braking issue was fixed. The last issue still occurs, but not every time. @CYBRLFT put it perfectly in a recent video, “had an issue for a while now” & “it’s the little things”
9/11

Tesla has fixed some issues & it’s exciting to see when it happens, but the pace & consistency of fixes may appear to be slow. This chart shows 300+ persistent issues fixed in the small group of community testers. This needs to get exponential with no regression.
10/11

Parting thoughts: It’s hard to measure progress. We should consider different perspectives. Improvements have been made & anyone testing daily for >6 months knows this. Tesla will need to fix a lot of little things & I look forward to cont. testing / measuring progress.
11/11