Santiago Profile picture
14 Mar, 9 tweets, 3 min read
A good way to understand how shit works is by breaking it down as much as you can.

Here is some code showing Dropout working on an array. And this is a thread explaining how it works.

☕️🧵👇
First, the code.

I want you to notice that Dropout does a couple of things:

▫️ It zeroes-out a percentage of the units.
▫️ It scales the remaining units to account for the missing values.

The second one wasn't obvious to me.
Remember those kids from school that sat together and copied from each other during exams?

They aced every test but were hardly brilliant, remember?

Eventually, the teacher had to set them apart. That was the only way to force them to learn.
The same happens with neural networks.

Sometimes, a few hidden nodes create associations that do most of the work, forcing the network to ignore the rest.

This is called co-adaptation, and it prevents networks from generalizing.
We can solve this problem like teachers do: breaking the associations that prevent the network from learning.

This is what Dropout is for.

During training, Dropout randomly removes some of the units.

This forces the network to learn in a balanced way.
Now every unit is on its own and can't rely on other units to do their work. They have to work harder by themselves.

Dropout works very well, and it's one of the main mechanisms to reduce overfitting.
Great question!

The remaining nodes will pick up the slack and learn.

If we keep dropping nodes, the ones remaining need to work harder and independently to learn.
This is a very good question!

Even if you start with a narrower network, there's no guarantee that nodes won't start collaborating and rendering useless the rest of the nodes. Dropout will ensure that doesn't happen.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Santiago

Santiago Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @svpino

15 Mar
You aren't doing yourself any favors if you aren't throwing away your validation data regularly.

It's painful, I know, but you are looking for trouble if you don't do it.

Let's talk about what happens with your data and your model.

Grab the ☕️, and let's do this thing. 🧵👇
Every machine learning tutorial teaches you about splitting your dataset.

They either go with train/test or train/validation/test. Nomenclature doesn't matter here. You just need to understand how each one of these is used.

Here is a thread about this:

Let's think of a neural network and focus on the train set for a second.

We use this to train our model. The data on this set is the one the network uses to adjust the weights.

And, of course, the model will get really good at solving this set.
Read 8 tweets
15 Mar
My recommendation to learn machine learning:

▫️ Machine Learning Crash Course (Google)
▫️ Machine Learning (Coursera)

Take them in order. They are both free. They are both amazing.

(Before you embark on this journey, make sure you feel comfortable writing Python 🐍.)
Not really. Nothing has changed with the fundamentals. The course is as relevant today as it was back in 2010.
Experience will help you make a few sensible choices that you can later test.
Read 8 tweets
12 Mar
Do you like Star Wars?

Let's talk about John, a machine learning engineer.

He is building a model to predict whether people will like a future Star Wars episode.

This is the story of how John screwed things up.

☕️🧵👇
They are screening the new Star War episode, and the theater seems like a good place to collect some information to create the model.

John goes right in and gives an optional survey to the people sitting in the middle of the theater right before the movie starts.
John does the same for a couple of weeks, then goes back to his computer and puts together a model.

It turns out that the model sucks.

The new episode doesn't do well despite the model predicting the opposite.

What happened?
Read 8 tweets
12 Mar
“Autoencoders and rotten bananas” is the first story trying to teach you something new about machine learning.

A lot of work to get this out, and I hope you enjoy it as much as I did writing it.

digest.underfitted.io/archive/364757
This is such a great question!

Why in the world do we need autoencoders when we have pretty good compression algorithms?

Autoencoders will never do better than our existing techniques (like for example, JPEG encoding).

JPEG works for any image. Autoencoders only work for the type of images that it was trained on.

That's a big disadvantage.

But only if you are trying to use them to compress images.

Here is a killer application of autoencoders: anomaly detection.
Read 5 tweets
11 Mar
Before you start building a machine learning model, you need a baseline.

I find it helpful to think about 3 different levels and tackle them in order.

Here is how I do this: ☕️🧵👇
▫️ Level 1: The human baseline

Before anything else happens, I find it useful to understand how humans do when solving the problem.

This gives me the ceiling that I should aspire to (or maybe even beat it, if I'm lucky!)
Sometimes, the human baseline will hint at whether a model is a feasible solution for the problem.

For example, if the data doesn't contain information that we can use to make predictions, humans will do very poorly. This will save us a lot of work!
Read 10 tweets
11 Mar
"Do I need a Ph.D. or a Master's degree to work as a machine learning engineer?"

No.

A lot of companies ask for degrees to weed out people that apply to jobs prematurely.

If you have the required skills and show your experience, the degree will not matter.
Both are valid paths and it will come down to your personal preference or what you need to accomplish

If you aren’t sure, online courses give you less risk upfront.
There are specific positions that do require (and will continue to do so) a degree in a related field.

But the industry has changed in the last few years. Fewer places require degrees anymore, and the tendency will continue in that direction.

Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!