Given the pushback from the AV industry on measuring progress with miles per disengagement, let’s talk about alternatives.
Inside @voyage, we measure our progress with something we call the Driverless Readiness Score (DRS).
How does it work?
Inside @voyage, we measure our progress with something we call the Driverless Readiness Score (DRS).
How does it work?
The DRS represents progress toward a set of key metrics and deliverables. When goal metrics and deliverables are hit, we are ready for driverless operations.
What’s important is that this bundle of metrics & deliverables are chosen _specifically_ for our targeted roadway.
What’s important is that this bundle of metrics & deliverables are chosen _specifically_ for our targeted roadway.
DRS = (Metrics Progress / 2) + (Deliverables Progress / 2)
Let’s talk about metrics. We track many metrics, but our primary KPIs account for 50% of our DRS.
✅ MPI
✅ MPD
✅ Miles per false negative in critical zones
✅ % of sim scenarios passing
All KPIs contribute equally, with goals dictated by our roadway requirements.
✅ MPI
✅ MPD
✅ Miles per false negative in critical zones
✅ % of sim scenarios passing
All KPIs contribute equally, with goals dictated by our roadway requirements.
But, metrics alone cannot always tell you if you are making progress, especially week-to-week.
Some deliverables are binary. Are you running on a safety-certifiable middleware? Does your vehicle have appropriate redundancies? Does X component have reliable diagnostics?
Some deliverables are binary. Are you running on a safety-certifiable middleware? Does your vehicle have appropriate redundancies? Does X component have reliable diagnostics?
We track deliverables with priority (P/1/2) and status (🔴🟡🟢), and each is assigned a progress score. The status of these deliverables changes week-to-week, and priority weights it heavier.
The sum of all these deliverables then makes up the remaining 50% of our DRS.
The sum of all these deliverables then makes up the remaining 50% of our DRS.
Interestingly, comparing our DRS to the DRS of another company would be an apples-to-apples comparison of readiness for driverless (no Test Driver!) in our respective targeted applications.
This is perhaps exactly what the CA DMV is looking for.
This is perhaps exactly what the CA DMV is looking for.
What’s more, I believe DRS serves as a great internal tool. As a small company, it’s critical to have a pulse that says if we are moving forward (or not). Every day matters.
We’re not ready just yet, but I’d love to open source DRS (and our score) as a template for all companies to report.
Given that people do not opt-in to be near AV testing, we cannot stand back and say that MPD is imperfect, while also not sharing alternatives.
Given that people do not opt-in to be near AV testing, we cannot stand back and say that MPD is imperfect, while also not sharing alternatives.
Feedback appreciated 🙏
