Thread by @olivercameron

🍎 I’ve seen a lot of headlines and charts about how far behind ’s self-driving car is (1.1 miles per disengagement!)

Join me as I try to explain why this is incorrect, and why  (according to their report) is perhaps in third place, as ranked by this terrible metric.

 took the weird step of reporting two metrics at different periods in 2018.

1️⃣ Raw disengagements (until June 2018)
2️⃣ Important disengagements (from July 2018 onward)

Any program that’s been around for some time just reports 2️⃣ in their CA DMV report. Never 1️⃣.

 reported _all_ disengagements from conception of their program until June 2018. Even if it was a planned disengagement during a structured test, they still reported it.

This is how you get to 1.1 miles per disengagement. No one else (AFAIK) reports _all_ disengagements.

The right metric to compare the different programs is “important” or “safety-critical” disengagements, which is the default metric reported these days.

If the Test Driver didn’t take over, would something dangerous have happened?

 switched to this disengagement “standard” (like everyone else) from July 2018, and reported impressive results...

’s results: 56,135 miles driven with 28 important disengagements.

That’s one disengagement every ~2,000 miles.

That places  in third place (begins Cruise).

There’s a lot of issues with comparing disengagement rates, because everyone decides for themselves exactly what an “important” disengagement is.

How can you know for sure a takeover was safety-critical? Especially if the Test Driver took over preemptively?

Some programs (like us) determine how important a disengagement was with re-sim. They generate a sim scenario with the same variables and see how the car would have handled it if not disengaged.

However, each program’s sim is very different, and doesn’t replicate all variables.

Other programs may just look at the raw drive data and assess manually. You can imagine this is prone to errors 😬

It’s also important to note: not all disengagements are created equally.

1 disengagement in a complex city center is not the same as 1 disengagement on a highway.

Given the variability, take any disengagement rates you read about with a hefty grain of salt. Use them as just one of many variables to assess progress of a program.

In conclusion: I’m not sure if an upheld disengagement standard will ever happen, but let’s applaud  for its transparency and progress. We’ll have safer roads because of them!

Read their cover letter here 👇

dmv.ca.gov/portal/wcm/con…