Share this page!

Jean de Nyandwi Profile picture
Twitter logo
21 Nov, 12 tweets, 4 min read
Machine Learning Weekly Highlights 💡

Made of:

◆2 things from me
◆2 from other creators
◆2+1 from the community

A thread 🧵
This week, I wrote about activation functions and why they are important components of neural networks.

Yesterday, I also wrote about image classification, one of the most important computer vision tasks.
#1

Here is the thread about activation functions

#2

Here is the thread about image classification.

From Other Creators

#1

A Review of latest cool papers

@omarsar0 made a great summary about the papers that he recently shared such as a survey of visual transformers, video object segmentation, neural rendering, Graph Neural Networks(GNNs), etc...

#2

Thinking in cycles

@RodOrtJose shared five great tips on doing machine learning the right way.

From the Community

#1

TensorFlow released a TensorFlow GNN, a new Graph Neural Network library that has Keras API component.

blog.tensorflow.org/2021/11/introd…
I am interested in learning about Geometric Deep Learning. In the near future, I will learn about it and will share it with you along the way...
#2

OpenAI made it easy to get access to GPT-3. In a matter of seconds, I just got this massive language model. If I get time in the coming weeks, I will explore it.

openai.com/blog/api-no-wa…
#3

NYU Deep Learning course(by ylecun & @alfcnz) materials are out.

cds.nyu.edu/deep-learning/

That's it from this week.

I would like to keep doing weekly highlights to share things that I found helpful over the whole week. For deep dives, I am starting a newsletter in next month.

Let me know if the highlights help you catch up with what's happened in the community.
Until the next week, stay safe!

And make sure you follow @Jeande_d for more machine learning ideas and the latest news.

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Jean de Nyandwi

Jean de Nyandwi Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Jeande_d

Jean de Nyandwi Profile picture
20 Nov
Image classification is one of the most common & important computer vision tasks.

In image classification, we are mainly identifying the category of a given image.

Let's talk more about this important task 🧵🧵
Image classification is about recognizing the specific category of the image from different categories.

Take an example: Given an image of a car, can you make a computer program to recognize if the image is a car?
One might ask why we even need to make computers recognize the images. He or she would be right.

Humans have an innate perception system. Identifying or recognizing the objects seems to be a trivial task for us.

But for computers, it's a different story. Why is that?
Read 15 tweets
Jean de Nyandwi Profile picture
17 Nov
Activations functions are one of the most important components of any typical neural network.

What exactly are activation functions, and why do we need to inject them into the neural network?

A thread 🧵🧵
Activations functions are basically mathematical functions that are used to introduce non linearities in the network.

Without an activation function, the neural network would behave like a linear classifier/regressor.
Or simply put, it would only be able to solve linear problems or those kinds of problems where the relationship between input and output can be mapped out easily because input and output change in a proportional manner.

Let me explain what I mean by that...
Read 27 tweets
Jean de Nyandwi Profile picture
14 Nov
Machine Learning Weekly Highlights 💡

◆3 things from me
◆2 things from other people and
◆2 from the community

🧵🧵
This week, I wrote about what to consider while choosing a machine learning model for a particular problem, early stopping which is one of the powerful regularization techniques, and what to know about the learning rate.

The next is their corresponding threads!
1. What to know about a model selection process...

Read 13 tweets
Jean de Nyandwi Profile picture
12 Nov
Learning rate is one of the most important hyperparameters to adjust well during the ML model training.

A high learning rate can speed up the training, but it can cause the model to diverge. A low rate can slow the training.

Here are different learning rate curves Image
A low learning rate can also give poor results.

A good recommended practice is to usually start with a high rate and then reduce it accordingly.

There are many techniques that can be used to achieve that. They are called learning rate schedulers.
Example of learning rate scheduling techniques:

◆Power scheduler
◆Exponential scheduler
◆Piecewise constant or multi-factor scheduler
◆Performance scheduler
◆Cosine schedule
Read 4 tweets
Jean de Nyandwi Profile picture
11 Nov
The initial loss value that you should expect to get when using softmax activation in the last layer of the neural network:

Initial loss = ln(number_of_classes), ln being a natural logarithm.
Example:

last_layer = api.layers.dense(10, activation='softmax')

# number of classes = 10
initial_loss = ln(10) #2.302

Understanding this is important when it comes to debugging the network. If you see a loss of 4.5 when you have 10 classes, there is something wrong.
Also, the reported loss on the first training epoch is the average loss of the whole batch.

Thus, you may instead get the initial loss less than ln(number_of_classes) because you are training in batches. And it is a good thing.
Read 4 tweets
Jean de Nyandwi Profile picture
10 Nov
The below illustration shows early stopping, one of the effective and simplest regularization techniques used in training neural networks.

A thread on the idea behind early stopping, why it works, and why you should always use it...🧵 Image
Usually, during training, the training loss will decrease gradually, and if everything goes well on the validation side, validation loss will decrease too.

When the validation loss hits the local minimum point, it will start to increase again. Which is a signal of overfitting. Image
How can we stop the training just right before the validation loss rise again? Or before the validation accuracy starts decreasing?

That's the motivation for early stopping.

With early stopping, we can stop the training when there are no improvements in the validation metrics. Image
Read 15 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Thank you for your support!

Follow Us on Twitter!

Like this author's thread?

Get emails when @Jeande_d has new unrolls!

Check out other threads from @Jeande_d!

Read all threads by @Jeande_d

:(