Thread by @Tweetermeyer

First of all, this gross utilitarianism (people will die but it will be worth it) has no place in the development of complex, safety-critical systems. If you aren't principled about safety, you won't get it. Just ask Uber ATG... oh wait, they don't exist anymore!

https://twitter.com/TESLAcharts/status/1442981125597630464

Uber ATG bought into this utilitarian logic, it cut corners on safety in the development process, and it killed an innocent human being. What it never saw was the implicit payoff: a technical advantage arising from its embrace of these deaths as inevitable and worth it.

Today, Uber ATG no longer exists. There are lots of lessons to be learned from their experience, but the most important part is that rushing and cutting corners on safety only causes innocent deaths and delivers no advantages: thedrive.com/tech/27023/10-…

One valuable thing that did come out of that tragedy, besides some sector-wide clarity about what works and what doesn't in this tech, was a safety case framework later adapted by @aurora_inno.

This is what a principled approach to safety looks like: aurora.tech/blog/safety-ca…

If Musk really wants to convince the public that he cares about the "reality of safety," in spite of his flouting of the sector's consensus on basic safety measures, he could show the work Tesla has done to measure and ensure safety in its driving automation. A safety case.

Ultimately, the "perception of safety" does matter here because AV developers must ask the public to trust their technology with their lives. Even if cutting corners (e.g. testing on the public) could deliver advantages (it won't), every death hurts public trust of all AVs.

Tesla's design decisions in, and communication about, driving automation technology have already contributed to multiple deaths, some of which were misreported as being "AV crashes." Musk has already done more to harm public trust in this technology than literally anyone else.

I don't think the industry wants a fight with Musk. I think most of the leaders in the AV sector would love to have him as a leader in, and advocate for the technology they are so passionate about. But it's this risk tolerance, this embrace of death, that makes Musk anathema.

The public faces a stark choice here: between one firm's utilitarianism that says anyone killed by an in-development AV is the cost of doing business, and a broad consensus that every crash, injury and death must be avoided in a principled way. Which would you trust your life to?