There are different computer vision problems you need to solve in a self-driving car.
βͺοΈ Object detection
βͺοΈ Lane detection
βͺοΈ Drivable space detection
βͺοΈ Semantic segmentation
βͺοΈ Depth estimation
βͺοΈ Visual odometry
Details π
Object Detection ππΆββοΈπ¦π
One of the most fundamental tasks - we need to know where other cars and people are, what signs, traffic lights and road markings need to be considered. Objects are identified by 2D or 3D bounding boxes.
Relevant methods: R-CNN, Fast(er) R-CNN, YOLO
Distance Estimation π
After you know what objects are present and where they are in the image, you need to know where they are in the 3D world.
Since the camera is a 2D sensor you need to first estimate the distance to the objects.
Relevant methods: Kalman Filter, Deep SORT
Lane Detection π£οΈ
Another critical information the car needs to know is where the lane boundaries are. You need to detect not only lane markings, but also curbs, grass edges etc.
There are different methods to do that - from traditional edge detection based methods to CNNs.
Driving Path Prediction ‴οΈ
An alternative is to train a neural network that will directly output the trajectory that the car needs to drive. This can be used as a substitute to centering between the lane markings if they are not visible for example.
Drivable Space Detection βοΈ
The goal here is to detect which parts of the image represent the space where the car can physically drive onto.
The methods here are usually very similar to the semantic segmentation methods (see below).
Semantic Segmentation π¨
Not all parts of the image can be described by a bounding box or a lane model, e.g. trees, buildings, the sky. Semantic segmentation methods classify each pixel in the image.
The goal is to estimate the distance to every pixel in the image, in order to have a better 3D model of the surrounding.
Methods like stereo and structure-from-motion are now being replaces by self-supervised deep learning models working on single images.
Visual Odometry π₯
While we know the movement of the car from the wheel sensors and IMU, determining the actual movement in the camera can be more accurate to get the pitch angle for example.
The visual odometry estimates the 6 DoF movement of the camera between two frames.
Summary π
There are of course many other computer vision problems that may be helpful, but this thread will give you an overview of the most important ones.
As you see, nowadays, deep learning methods (and especially CNNs) dominate all aspects of computer vision...
If you liked this thread and want to read more about self-driving cars and machine learning follow me @haltakov!
I have many more threads like this planned π
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
How can I prove to you that I know a secret, without revealing any information about the secret itself?
This is called a zero-knowledge proof and it is a super interesting area of cryptography! But how does it work?
Thread π§΅
Let's start with an example
Peggie and Victor travel between cities A and B. There are two paths - a long path and a short path. The problem is that there is a gate on the short path for which you need a password.
Peggie knows the password, but Victor doesn't.
π
Victor wants to buy the password from Peggie so he can use the short path.
But what if Victor pays Peggie, but she lied and she didn't know the password? How can Peggie prove to Victor she knows the password, without actually revealing it?
Rescue Toadz looks like a regular NFT collection at first - you can mint a toad and you get an NFT in your wallet.
100% of the mint fee is directly sent to @Unchainfund - an organization that provides humanitarian aid to Ukraine and that has already raised $9M!
π
@ianbydesign@RescueToadz@Unchainfund@cryptoadzNFT The process is completely trustless and automatic! All the logic is coded in the smart contract which cannot be changed and which everybody can inspect.
You trust the code, not us! We have no way to steal the funds even if we wanted (we don't π).
Principal Component Analysis is a commonly used method for dimensionality reduction.
It's a good example of how fairly complex math can have an intuitive explanation and be easy to use in practice.
Let's start from the application of PCA π
Dimensionality Reduction
This is one of the common uses of PCA in machine learning.
Imagine you want to predict house prices. You get a large table of many houses and different features for them like size, number of rooms, location, age, etc.
Some features seem correlated π
Correlated features
For example, the size of the house is correlated with the number of rooms. Bigger houses tend to have more rooms.
Another example could be the age and the year the house was built - they give us pretty much the same information.
For regression problems you can use one of several loss functions:
βͺοΈ MSE
βͺοΈ MAE
βͺοΈ Huber loss
But which one is best? When should you prefer one instead of the other?
Thread π§΅
Let's first quickly recap what each of the loss functions does. After that, we can compare them and see the differences based on some examples.
π
Mean Square Error (MSE)
For every sample, MSE takes the difference between the ground truth and the model's prediction and computes its square. Then, the average over all samples is computed.