Thread by @paulabartabajo_

How do you know if your Machine Learning model is "good enough", in a real-world project? 🤔

Let me explain ↓

A Machine Learning model in the real world is either:

✅ Good enough. In this case, the model is deployed, its predictions are used, and add value to the business every day.

❌ Not good enough. You need to improve your model, and if that is not possible, the project is canceled

When developing a model, you use standard metrics to measure how good it is.

Examples:
→ Mean Square Error → regression problems.
→ Accuracy → classification problems.

These are the metrics you care about in Kaggle.

However, in real-world projects, these are insufficient.

In real-world ML, you need to translate these "abstract" metrics into "business" metrics, that everyone in the team/company understands.

And the goodness of your ML model is judged based on these "business" metrics.

This is the baseline you need to beat.

#example

Imagine you work at Tesla, building the next generation of self-driving cars.

You wanna build a better version of the autopilot system, which decides in real-time what the car should do next. For example:

1. go straight ⬆️
2. turn left ⬅️
3. turn right ➡️
4. stop ✋

Imagine you have historical data with labels you can use to train your ML model, in this case, a classifier with 4 possible outcomes.

And you manage to build a model with 99% accuracy.

The question is: is this accuracy "good enough", or not?

To answer this, you need to translate this abstract "accuracy" into something meaningful for the business.

For example, "probability of a car crash"

To greenlight your new autopilot system, the team needs to ensure that the implied "probability of a car crash" is

→ lower than the current system's (baseline 1)

→ lower than the probability of a crash when a human drives the car (baseline 2)

To map your "abstract" accuracy of 99%, into a meaningful "probability of a car crash", you immerse your ML agent into a simulation engine and let it navigate it as much as possible.

Every time the agent crashes, you record it, and you get your crash metric.

You compare this metric with the 2 baselines, and decide the model is "good enough" when

(your_system_crash < baseline 1) AND (your_system_crash < baseline 2)

If either one of these inequalities does not hold, the model is NOT good enough, and you need to work on it further.

To sum up,

→ Real-world ML models are ultimately evaluated in terms of business metrics.

→ An ML model is "good enough" when its implied business metric beats the status quo.

Wanna get more real-world ML content?

Subscribe to my newsletter and get for FREE my eBook

"How to become a freelance data scientist"

which has specific advice to help you become a freelance data scientist ↓
datamachines.xyz/subscribe/

Wanna build a real-world ML project and rock your next job interview?

Check the NEW mentorship program ↓
datamachines.xyz/data-science-m…

That's all for now folks.

I hope you find this content useful for your next ML project 💪🏽

Wanna connect? ↓
Follow me @paulabartabajo_

Wanna help?
Like/Retweet the first tweet below to spread the wisdom
↓↓↓

https://twitter.com/paulabartabajo_/status/1569665691426648066