Thread by @haltakov

Is your machine learning model performing well? What about in 6 months? 🤔

If you are wondering why I'm asking this, you need to learn about 𝗰𝗼𝗻𝗰𝗲𝗽𝘁 𝗱𝗿𝗶𝗳𝘁 and 𝗱𝗮𝘁𝗮 𝗱𝗿𝗶𝗳𝘁.

Let me explain this to you using two real world examples.

Thread 👇

Imagine you are developing a model for a self-driving car to detect other vehicles at night.

Well, this is not too difficult, since vehicles have two red tail lights and it is easy to get a lot of data. You model works great!

But then... 👇

Car companies decide to experiment with red horizontal bars instead of two individual lights.

Now your model fails to detect these cars because it has never seen this kind of tail light.

Your model is suffering from 𝗰𝗼𝗻𝗰𝗲𝗽𝘁 𝗱𝗿𝗶𝗳𝘁

👇

Concept drift happens when the objects you are trying to model change over time.

In the case above, cars changed and you now need to adapt your model.

Another example 👇

You are now dealing with the detection of traffic sights. Again, at night things are pretty easy because signs reflect the lights from the car's headlights and are very well visible in the image. Easy!

And again something happens... 👇

New cars start getting more powerful laser high beams and now suddenly the signs reflect so much light that there are overexposed in the image.

The problem now is 𝗱𝗮𝘁𝗮 𝗱𝗿𝗶𝗳𝘁.

👇

Data drift happens when the object you are modeling stays the same, but the environment changes in a way that affects how the object is perceived.

In this case, the signs are exactly the same, but they appear different because of the lighting.

👇

Since we talked about Sampling Bias last week, we can connect it with the concept of drift. Since the underlying distribution of your data changed, even if your sampling was good in the beginning it isn't anymore

You now have a Sampling Bias problem

https://twitter.com/haltakov/status/1494389686382612489

👇

What can we do about it?

While there are some methods like online learning to continue improving the model and some others to detect the drift, usually the solution is just to retrain your model.

This is something you need to be prepared about in your pipeline!

👇

Follow me @haltakov for more intuitive explanations of machine learning and web 3 topics.

This depends on your application and how much error you can tolerate. You first need to monitor the model and detect that it is getting worse. Then collect data and retrain.

You can also keep the data collection ongoing and retrain periodically.

https://twitter.com/Ali_k2066/status/1497068087488782336?s=20&t=Vb9nB5YMoupJM8QYmTM9Yg