Thread by @Sachintukumar

Day 58 of #100DayswithMachineLearning

Topic - Stochastic Gradient Descent ( SGD )

A Thread 🧵

SGD is an optimization algorithm often used in machine learning applications to find the model parameters that correspond to the best fit between predicted and actual outputs. It’s an inexact but powerful technique.

Saddle point or minimax point is point on the surface of graph of function where slopes (derivatives) in orthogonal directions are all zero (a critical point), but which is not local extremum of function

A saddle point (in red) on graph of z = x2 − y2 (hyperbolic paraboloid)

Cost or loss function is function to be minimized (or maximized) by varying decision variable

They tend to minimize diff between actual & predicted output by adjusting model parameters(like weights & biases for neural network decision rules for random forest or gradient boosting

Stochastic Gradient Descent (SGD) is a variant of the Gradient Descent algorithm used for optimizing machine learning models. In this variant, only one random training example is used to calculate the gradient and update the parameters at each iteration

Advantages:

Speed: SGD is faster than other variants of Gradient Descent such as Batch Gradient Descent and Mini-Batch Gradient Descent since it uses only one example to update the parameters.

Memory Efficiency

Avoidance of Local Minima

Disadvantages:

Noisy updates: The updates in SGD are noisy and have a high variance, which can make the optimization process less stable and lead to oscillations around the minimum.

Slow Convergence

Sensitivity to Learning Rate

Less Accurate

This cycle of taking the values and adjusting them based on different parameters in order to reduce the loss function is called back-propagation
@geeksforgeeks

geeksforgeeks.org/ml-stochastic-…

Mini-Batch Gradient Descent: Parameters are updated after computing the gradient of  the error with respect to a subset of the training set

@github Notebook

github.com/sachinkumar160…

📷If this thread was helpful to you

1. Follow me @Sachintukumar
for daily content

2. Connect with me on Linkedin: linkedin.com/in/sachintukum…

3. RT tweet below to share it with your friend