Self-driving car engineer roles - Computer Vision Engineer π
The camera is one of the most important sensors! It is not always the most accurate one, but it can provide much more data than a lidar or radar. Extracting this data is the job of the CV engineer.
Thread π
Problems to work on π€
Here some typical object classes that need to be detected and classified.
π Traffic signs
π¦ Traffic lights
π Vehicles
πΆ Pedestrians
π¦ Animals
π£οΈ Lane markings
ποΈ Landmarks
π§ Construction zones
𧱠Obstacles
π Police cars
Distance estimation π
Detecting an object is not enough, though. You also want to know how far the object is from the car. While the detection part is dominated by deep learning, the traditional CV methods (e.g. Kalman Filter) are still very useful for distance estimation.
Semantic segmentation π¨
Semantic segmentation is useful for detecting objects and areas in the image that don't have a well defined shape, like buildings, trees, the free space on the road and so on. It can also serve as a basis for object detection algorithms.
Depth estimation π
A self-driving car needs a good 3D model of the surrounding. While lidar (and to some extent radar) can provide that, the camera can be seen as a redundant sensor.
There are different approaches:
- Stereo cameras
- Structure-from-motion
- Single frame depth
Relevance βοΈ
Another important aspect is how relevant is an object:
- Is the car in my lane or not?
- Is the red or the green traffic light relevant for my lane?
- Is the pedestrian going to jump on the road in front of the car?
Here, fusion with other sensors can help as well.
Required knowledge π
Nowadays, deep learning and CNNs are a must for a computer vision engineer. However, don't underestimate the importance of traditional CV methods, especially around tracking, distance estimation and depth estimation!
Programming language π»
Computer vision is very computationally intensive, so good C++ skills are required. However, for training and evaluation CNNs you usually only need Python!
Getting started ποΈ
There are again many ways to enter the industry as a CV engineer. Pick one of the problems above and focus on it to gain knowledge and experience!
Self-driving car engineer roles - Software Engineer π»
There are many specialized roles in a self-driving car project, like ML or CV engineers. However, every projects needs lots of good software devs - you can enter the industry even without specific knowledge!
Thread π
Problems to work on π€
Some problems that software developers work on to build a self-driving car (the list is not exhaustive):
- HMI
- Operating system
- Logging and tracing
- Communication between ECUs
- Internal frameworks and libraries
- Implementing diagnostic interfaces
Software engineers also work closely with many of the more specialized roles.
For example with Machine Learning engineers to implement models on the ECU or with Vehicle Control engineers to get their algos working efficiently.
Some interesting self-driving news lately:
- Waymo launching test fleet without safety driver
- Tesla launching a beta of their Full Self-Driving
- Mercedes announcing a level 3 traffic jam pilot for 2021
There are 3 very different approaches π
1οΈβ£ "Everything that fits" approach.
This is Waymo's approach, but other companies like Cruise, Argo, Aurora, Uber, Zoox have a similar strategy.
Fit as much sensors as possible on the car, build high-definition maps of the environment and throw in lots of compute power.
Check out some images of these cars - they all have multiple lidars, cameras and radars all around the car. Waymo now has 29 cameras! π²
They are not really integrated in a consumer oriented way, but it should be fine for a robotaxi.
How to become a self-driving car engineer? π»π§ π
Well, there is no easy answer - building a self-driving car requires expertise from many different fields.
The good new is that there are many paths that will lead you to a job in this industry!
Read more details below π
There are many different engineering roles in a self-driving car project:
π» Software Development
π§ Machine Learning
π Computer Vision
ποΈ Vehicle Control
βοΈ Hardware
π½ Big Data
πΉοΈ Simulation
π Mapping
π§ Safety
π Security
π Test and Validation
π¨ Tooling
and more... π
Over the next days I'll share some of my experience working in the industry and describe in more details the different roles and what skills are required to enter the field.
A lot of the technology behind the FSD software was presented at a conference in February. Check out this video, it is really interesting!
Main point is Tesla is using a "Bird's eye view" neural network to predict the layout of the road.
You can see this in action in the car - you see all lines defining the lanes and the road boundaries in the display. They seem to be coming from the vision system and not from a map, because they are very unstable and often not very accurate.