Thread by @olivercameron

🧠 @Waymo published a paper on a model called ChauffeurNet, which attempts to drive a car using imitation learning.

ChauffeurNet aspires to more human-like driving, in contrast to traditional robotics approaches which drive like...well...a robot.

sites.google.com/view/waymo-lea…

Typical end-to-end approaches (raw sensor input to output raw control commands) struggle when faced with traffic rules (stop signs, traffic lights, etc.). ChauffeurNet aims to make progress with that problem to produce a more human-like driver that can handle complex scenarios.

ChauffeurNet takes a different approach than typical end-to-end approaches. There's a separate perception stack that processes sensors output (LIDAR & cameras), which is then fed into the model as an input, along with predictions of objects and a pre-computed route.

ChauffeurNet does use a map, but a very different map than traditional self-driving cars. Think more Google Maps than a point cloud of the world.

Because of these approaches (perception primarily), ChauffeurNet is not comparable to end-to-end, camera-only models you've heard of. Think of it as more of a potential replacement for a traditional motion planning module.

In a traditional planning module, you'll have a number of algorithms that dictate the exact path of a vehicle along a route, along with the speed and distance relative to other vehicles.

ChauffeurNet tries to accomplish that in a model, by imitating human examples.

ChauffeurNet (a RNN) is trained on 60 days of real-world expert driving. They attempt to demonstrate in both simulation and the real-world:

✅ Recovery from a bad path
✅ Nudging around parked cars
✅ Slowing down for slow car
✅ Lane curve following
✅ Stop sign and turn

They found that, with their first approach, pure imitation learning with 30 million examples was not sufficient. They often found the (simulated) car would collide with other vehicles or get stuck.

However, they improved model performance dramatically with "imitation dropout", exposing the model to additional behaviors such as off-road driving & collisions.

Here's that same nudge. The new model has a 10% collision rate (vs. 50%). Some caveats to that #, so read the paper!

They also tried ChauffeurNet on a @Waymo car (presumably at Castle, their testing grounds). Going from simulation to real-world is tough.

The conclusion from the ChauffeurNet team?

"The model is not yet fully competitive with motion planning approaches but we feel that this is a good step forward for machine learned driving models."

👍 This is impressive work from @Waymo. I think it goes to show just how much exploration there is still to do.

The above was a very brief introduction to ChauffeurNet, so I highly recommend reading the paper in its entirety here (HT to @amir).

arxiv.org/abs/1812.03079