Self-driving car engineer roles - Computer Vision Engineer πŸ‘€

The camera is one of the most important sensors! It is not always the most accurate one, but it can provide much more data than a lidar or radar. Extracting this data is the job of the CV engineer.

Thread πŸ‘‡
Problems to work on πŸ€”

Here some typical object classes that need to be detected and classified.

πŸ›‘ Traffic signs
🚦 Traffic lights
🚘 Vehicles
🚢 Pedestrians
🦌 Animals
πŸ›£οΈ Lane markings
πŸ”οΈ Landmarks
🚧 Construction zones
🧱 Obstacles
πŸš” Police cars
Distance estimation πŸ“

Detecting an object is not enough, though. You also want to know how far the object is from the car. While the detection part is dominated by deep learning, the traditional CV methods (e.g. Kalman Filter) are still very useful for distance estimation.
Semantic segmentation 🎨

Semantic segmentation is useful for detecting objects and areas in the image that don't have a well defined shape, like buildings, trees, the free space on the road and so on. It can also serve as a basis for object detection algorithms.
Depth estimation πŸ”­

A self-driving car needs a good 3D model of the surrounding. While lidar (and to some extent radar) can provide that, the camera can be seen as a redundant sensor.

There are different approaches:
- Stereo cameras
- Structure-from-motion
- Single frame depth
Relevance ⁉️

Another important aspect is how relevant is an object:
- Is the car in my lane or not?
- Is the red or the green traffic light relevant for my lane?
- Is the pedestrian going to jump on the road in front of the car?

Here, fusion with other sensors can help as well.
Required knowledge πŸ“š

Nowadays, deep learning and CNNs are a must for a computer vision engineer. However, don't underestimate the importance of traditional CV methods, especially around tracking, distance estimation and depth estimation!
Programming language πŸ’»

Computer vision is very computationally intensive, so good C++ skills are required. However, for training and evaluation CNNs you usually only need Python!
Getting started 🏎️

There are again many ways to enter the industry as a CV engineer. Pick one of the problems above and focus on it to gain knowledge and experience!

Example project - benchmark.ini.rub.de/?section=gtsrb…

Check below for some free learning resources πŸ‘‡
Resources πŸ“–

Check out the following resources to start learning:
- Deep Learning for CV: coursera.org/learn/convolut…
- Classical CV: udacity.com/course/introdu…
- TensorFlow: tensorflow.org/resources/lear…
- PyTorch: coursera.org/learn/deep-neu…
- OpenCV: docs.opencv.org/master/d9/df8/…

β€’ β€’ β€’

Missing some Tweet in this thread? You can try to force a refresh
γ€€

Keep Current with Vladimir Haltakov

Vladimir Haltakov Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @haltakov

27 Oct
Self-driving car engineer roles - Software Engineer πŸ’»

There are many specialized roles in a self-driving car project, like ML or CV engineers. However, every projects needs lots of good software devs - you can enter the industry even without specific knowledge!

Thread πŸ‘‡
Problems to work on πŸ€”

Some problems that software developers work on to build a self-driving car (the list is not exhaustive):
- HMI
- Operating system
- Logging and tracing
- Communication between ECUs
- Internal frameworks and libraries
- Implementing diagnostic interfaces
Software engineers also work closely with many of the more specialized roles.

For example with Machine Learning engineers to implement models on the ECU or with Vehicle Control engineers to get their algos working efficiently.

And tooling and testing are huge separate topic!
Read 8 tweets
27 Oct
Self-driving car approaches πŸ§ πŸš—

Some interesting self-driving news lately:
- Waymo launching test fleet without safety driver
- Tesla launching a beta of their Full Self-Driving
- Mercedes announcing a level 3 traffic jam pilot for 2021

There are 3 very different approaches πŸ‘‡
1️⃣ "Everything that fits" approach.

This is Waymo's approach, but other companies like Cruise, Argo, Aurora, Uber, Zoox have a similar strategy.

Fit as much sensors as possible on the car, build high-definition maps of the environment and throw in lots of compute power.
Check out some images of these cars - they all have multiple lidars, cameras and radars all around the car. Waymo now has 29 cameras! 😲

They are not really integrated in a consumer oriented way, but it should be fine for a robotaxi.
Read 12 tweets
26 Oct
How to become a self-driving car engineer? πŸ’»πŸ§ πŸš—

Well, there is no easy answer - building a self-driving car requires expertise from many different fields.

The good new is that there are many paths that will lead you to a job in this industry!

Read more details below πŸ‘‡
There are many different engineering roles in a self-driving car project:

πŸ’» Software Development
🧠 Machine Learning
πŸ‘€ Computer Vision
🏎️ Vehicle Control
βš™οΈ Hardware
πŸ’½ Big Data
πŸ•ΉοΈ Simulation
πŸ“ Mapping
🚧 Safety
πŸ”’ Security
πŸ“ Test and Validation
πŸ”¨ Tooling

and more... πŸ‘‡
Over the next days I'll share some of my experience working in the industry and describe in more details the different roles and what skills are required to enter the field.

I will link all threads below so stay tuned πŸ‘‡
Read 5 tweets
26 Oct
Tesla released a beta version of their Full Self Driving (FSD) software to selected people couple of days ago πŸ§ πŸš—.

The first impressions are quite interesting! Here some comments from my side.

Thread πŸ‘‡

Technology πŸ€–

A lot of the technology behind the FSD software was presented at a conference in February. Check out this video, it is really interesting!



Main point is Tesla is using a "Bird's eye view" neural network to predict the layout of the road.
You can see this in action in the car - you see all lines defining the lanes and the road boundaries in the display. They seem to be coming from the vision system and not from a map, because they are very unstable and often not very accurate. ImageImage
Read 13 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!