Before you start building a machine learning model, you need a baseline.
I find it helpful to think about 3 different levels and tackle them in order.
Here is how I do this: ☕️🧵👇
I find it helpful to think about 3 different levels and tackle them in order.
Here is how I do this: ☕️🧵👇
▫️ Level 1: The human baseline
Before anything else happens, I find it useful to understand how humans do when solving the problem.
This gives me the ceiling that I should aspire to (or maybe even beat it, if I'm lucky!)
Before anything else happens, I find it useful to understand how humans do when solving the problem.
This gives me the ceiling that I should aspire to (or maybe even beat it, if I'm lucky!)
Sometimes, the human baseline will hint at whether a model is a feasible solution for the problem.
For example, if the data doesn't contain information that we can use to make predictions, humans will do very poorly. This will save us a lot of work!
For example, if the data doesn't contain information that we can use to make predictions, humans will do very poorly. This will save us a lot of work!
To get a human baseline, you can annotate the data twice and treat one set of results as the ground-truth, and the other set as the predictions.
▫️ Level 2: The data-independent baseline
Now that we have an idea of how humans do with the problem, the next step is to build a straightforward baseline model that's completely independent of the data.
The idea is for the baseline model to simply guess the answers.
Now that we have an idea of how humans do with the problem, the next step is to build a straightforward baseline model that's completely independent of the data.
The idea is for the baseline model to simply guess the answers.
A couple of examples of independent baselines:
▫️ Random guessing the answer
▫️ Always returning the same answer
I like this baseline because I can put it together very quickly and it's the first step to measure any models.
▫️ Random guessing the answer
▫️ Always returning the same answer
I like this baseline because I can put it together very quickly and it's the first step to measure any models.
▫️ Level 3: The model baseline
After I know how random guessing does, I put together the simplest possible model that can beat it.
The idea here is not to be fancy but focus on getting better than the level 2 baseline with very little work.
After I know how random guessing does, I put together the simplest possible model that can beat it.
The idea here is not to be fancy but focus on getting better than the level 2 baseline with very little work.
This baseline will tell us whether there's enough signal in the data to build a model with predictive capabilities.
After we have one, our goal is to beat it with a better model.
After we have one, our goal is to beat it with a better model.
I like to keep replacing this level 3 baseline.
Every time I build a better model, it becomes the new yard-stick that I use to measure future progress.
Beating the baseline becomes a cool game to play.
Every time I build a better model, it becomes the new yard-stick that I use to measure future progress.
Beating the baseline becomes a cool game to play.
Hey, I know sometimes machine learning sucks.
But we can do this shit together!
Follow me, and every week I’ll help you navigate this thing from doing it, failing at it, learning, and fixing it.
🦕
But we can do this shit together!
Follow me, and every week I’ll help you navigate this thing from doing it, failing at it, learning, and fixing it.
🦕
• • •
Missing some Tweet in this thread? You can try to
force a refresh
