Thread by @levikul09

Gradient descent is a powerful optimization technique.

But it has 3 main types. What are the differences? When to use them?

Let's figure it out! 🧵

The 3 gradient descent types differ in how much data they use in the process.

This will result in a trade-off between accuracy and computation time.

These are really important aspects of an analysis, so using the appropriate gradient descent method is essential.

1️⃣ Batch gradient descent

This method uses the entire dataset to perform an update in the model.

With large data it can be very slow, but if the data is manageable, this method will provide a smooth convergence.

2️⃣ Stochastic gradient descent

In SGD one datapoint is used to take one step at a time.

This method can cause the optimization process to be less stable since its noisiness.

But SGD is computationally more efficient.

3️⃣ Mini-Batch Gradient Descent

A fixed number of training samples is used here (not only one but not all).

It tries to combine the advantages of the above-mentioned methods.

The only issue with Mini-Batch is that you have an additional parameter, the batch size.

That's it for today.

I hope you've found this thread helpful.

Like/Retweet the first tweet below for support and follow @levikul09 for more Data Science threads.

Thanks 😉

https://twitter.com/2712332320/status/1733067700149350737