Look at this video from a @Tesla Model 3 driving on the highway. The display shows multiple traffic lights coming out of the truck in front towards the car. What's going on? ๐ค
This is a typical case of a ๐๐ฟ๐ฎ๐ฐ๐ธ ๐น๐ผ๐๐!
Thread ๐
The problem ๐ค
The truck in front carries 3 real traffic lights. The problem is that the computer vision system on the Tesla assumes that traffic lights are static (which is a good assumption in general ๐). In this case, though, the traffic lights are moving at 120 km/h...
๐
Object detection ๐ฆ
A typical object detection system takes a single camera frame and detects all kinds of objects in it.
One of the best models for object detection is YOLO. I just ran this image through it and sure enough, it detects 2 of the traffic lights!
๐
If you want to learn more about how deep learning based object detection works, check out this thread.
We now want to track an object over multiple frames. In this way, we can more accurately determine the position of the object.
Given the detections in two frames, we need to associate each object in the first frame to an object in the second frame.
๐
Static object assumption ๐ฆ
For traffic lights, it is reasonable to assume they are static. Therefore, the change of its position in the second frame only depends on the speed of our own car.
Since we know it, we can predict where the light will be in the next frame.
๐
Track loss ๐ค
The problem is that these ๐ฆ are actually moving, so the prediction is wrong. The Tesla cannot associate them anymore, so:
1๏ธโฃ The light from the previous frame is counted as lost
2๏ธโฃ The light from this frame is regarded as a new object seen for the first time
๐
Tracking ๐
What happens to the lost object? The Tesla assumes the light can't be detected for some reason. Instead, It shows on the display the expected position of the light based on the vehicle speed.
That's why the lights come "flying" towards the car in the display.
As you can see in the video, this happens all the time. This is what happens when we make assumptions about the real world, that fail in some situations... ๐คทโโ๏ธ
The story is about police dogs trained to sniff drugs. The problem is that the dogs often signal drugs even if there are none. Then innocent people land in jail for days.
The cops even joke about the โprobable cause on four legsโ.
Let's see why is that ๐
1. Sampling Bias ๐ค
Drugs were found in 64% of the cars Karma identified, which was praised by the police as very good. In the end, most people don't carry drugs in their cars, so 64% seems solid.
There are different computer vision problems you need to solve in a self-driving car.
โช๏ธ Object detection
โช๏ธ Lane detection
โช๏ธ Drivable space detection
โช๏ธ Semantic segmentation
โช๏ธ Depth estimation
โช๏ธ Visual odometry
Details ๐
Object Detection ๐๐ถโโ๏ธ๐ฆ๐
One of the most fundamental tasks - we need to know where other cars and people are, what signs, traffic lights and road markings need to be considered. Objects are identified by 2D or 3D bounding boxes.
Relevant methods: R-CNN, Fast(er) R-CNN, YOLO
Distance Estimation ๐
After you know what objects are present and where they are in the image, you need to know where they are in the 3D world.
Since the camera is a 2D sensor you need to first estimate the distance to the objects.
Interesting results from the small experiment... ๐
This was actually a study reported in a Nature paper. Most people offer additive solutions (adding bricks) instead of substractive solutions (removing the pillar).
In this example, the most elegant solution is to remove the pillar completely and let the roof lie on the block. It will be simpler, more stable and won't cost anything.
Some people quickly dismiss this option assuming this is not allowed, but it actualy is ๐
This isn't because people don't recognize the value, but because many don't consider the substractive solution at all. Me included ๐โโ๏ธ
The paper shows that this happens a lot in real life, especially in regulation. People tend to add new rules, instead of removing old ones.
End-to-end approach to self-driving ๐ฅ ๐ธ๏ธ ๐น๏ธ
I recently wrote about the classical software architecture for a self-driving car. The end-to-end approach is an interesting alternative.
The idea is to go directly from images to the control commands.
Let me tell you more... ๐
This approach is actually very old, dating back to 1989 and the ALVINN model by CMU. It is a 3-layer neural network using camera images and a laser range finder.
A modern example is Nvidia's PilotNet - a Convolutional Neural Network with 250M parameters, which takes as input the raw camera image and predicts directly the steering angle of the car.
No explicit lane boundary or freespace detection needed!