Thread by @capeandcode

Gradient Descent is great but there are a whole bunch of problems associated with it.
Getting stuck in the local minima while browsing the solution space is one of the major issues.

A possible Solution?

SIMULATED ANNEALING

Here's a little something about it 🧵👇

The method of Simulated Annealing in Optimization is analogical to the process of Annealing in Metallurgy ⚗️🔥, hence the name.

We get stuck in the local minima because we tend to always accept a solution that seems best in shortsight. We just move in the downwards direction ⬇️ (negative gradient) and not upwards⬆️

So once we reach a point which is low but not the lowest, we may end up getting stuck.

🔹Either we can just stay there and accept current state as the solution, which is not very effective

🔹We can temporarily take a step in upwards direction that can get us out of the local minimum

And obvious enough we go with the second option.

To get out of local minimum we consider a random step in a direction.

If it improves the solution we always accept it. If not then we accept it anyways with a certain probability.

So we are accepting a bad move temporarily but it would make things better in the long run

T, the temperature 🌡️ parameter determines how bad a move are we willing to accept.

The higher the temperature, the more likely we are to accept a bad solution or a step upwards and vice-versa.

It doesn't end there, we also need a schedule for decreasing the value of the temperature parameter.

Can you guess the reason❓

To make sure that when we do reach near the global minimum solution we want to be less likely to accept worse solutions

There are multiple methods we can use to schedule the temperature reduction

This is how SIMULATED ANNEALING helps Gradient Descent to be better than usual.

Hope you enjoyed this!

We will see how to overcome some more problems of Gradient Descent.