Thread by @svpino

You aren't doing yourself any favors if you aren't throwing away your validation data regularly.

It's painful, I know, but you are looking for trouble if you don't do it.

Let's talk about what happens with your data and your model.

Grab the ☕️, and let's do this thing. 🧵👇

Every machine learning tutorial teaches you about splitting your dataset.

They either go with train/test or train/validation/test. Nomenclature doesn't matter here. You just need to understand how each one of these is used.

Here is a thread about this:

https://twitter.com/svpino/status/1359827674097672195?s=20

Let's think of a neural network and focus on the train set for a second.

We use this to train our model. The data on this set is the one the network uses to adjust the weights.

And, of course, the model will get really good at solving this set.

The validation set is just for that: we use it to compute our loss and maybe accuracy, so we know how good the model is.

Keep in mind the model is never trained on a validation set. No weights are optimized because of the data on this set.

At least not directly.

Here is the problem: we look at the model results on the validation set and change the model.

Yes, we may not be adjusting weights directly based on the data, but we are adjusting them indirectly by updating the model based on the results.

Every time you update your model based on your validation set results, you are leaking information and overfitting to that data.

It only takes a few rounds for the results on that set to be completely useless.

How do you avoid this? Ideally, you swap out that validation set as frequently as you can.

You can take all that data, merge it with the train data, and get a new fresh set.

Unfortunately, there's no real substitute for a fresh validation dataset to test your model.

Yes, there are different cross-validation techniques, but they won't fully solve the issue.

Bottom line: yeah, you probably need more data.