Thread by @edkesuma

This story took 3 months to create

A Deep Learning beginner to top 1% on Kaggle's latest Computer Vision competition

Here's my journey 👇

Competitions are a great place to learn from!

And beginners actually get the most value out of learning from comps.

If you're one of them, don't be intimidated and jump right in.

Humble beginnings

I first started learning DL when the competition started - October.

I picked up the amazing "Deep Learning with Pytorch" book.

Here, I learned torch basics (Part 1 of the book), including:

🕗 Tensors
🕗 Training / Inference
🕗 Neural nets from scratch
🕗 Autograd
🕗 Datasets and DataLoaders
🕗 Convolutions and pooling

Cool names right? 🤭

Joining the competition

After a month of building my basics, I jumped into the competition with a discussion post.

Many kagglers were helpful and their best advice was to read public notebooks.

So i went and read dozens of public notebooks.

While doing so, I got exposed to many libraries and ways of building the model.

My most used libraries ended up being:

🕗 timm
🕗 fastai
🕗 garbage collector
🕗 albumentations

Here's the biggest tip for creating models for deep learning competitions.

Fine.
Tune.
Pretrained models.

They give you a model that has seen plenty of images already.

And you only need to train it to fit your needs.

What's left to learn is how to do that fine tuning well.

Some useful ideas are:

🕗 Image augmentation
🕗 Mixup / Cutmix
🕗 Test time augmentation
🕗 Cyclical learning rate

Teaming up

Apes apart weak.
Apes together strong.

Doing the competition solo, I only spent time improving single models.

My team introduced the notion of model diversity.

This way, we get better ensembles/combinations of models.

These are the models my team used:

🕗 Biet
🕗 Volo
🕗 Swin
🕗 ConvNet

Those fine tuned models made their OOF predictions

And we used RidgeCV to find the weights to get the best ensemble

Future Improvements

I would do experiment tracking in the future.

Using any loggers like WandB or Neptune will help in summarizing experiment results to teammates.

Recap:
🕗 Solidify basics
🕗 Read public notebooks
🕗 Learn to fine tune well
🕗 Model diversity
🕗 Ensemble
🕗 Trust OOF score

Thanks for reading this far.

I tweet on DS/ML: threads, analogies, memes.

If you like this stuff, follow @edkesuma for more 😉

Happy Kaggling!

Additional: I documented the entire journey day by day on #66daysofdata.

Throwback to day one here 😆

https://twitter.com/edkesuma/status/1446440826838552580?t=l9qvbpUFlTA_Yk1uo0W_Rw&s=19