Gradient Descent is great but there are a whole bunch of problems associated with it.
Getting stuck in the local minima while browsing the solution space is one of the major issues.
A possible Solution?
SIMULATED ANNEALING
Here's a little something about it 🧵👇
The method of Simulated Annealing in Optimization is analogical to the process of Annealing in Metallurgy ⚗️🔥, hence the name.
We get stuck in the local minima because we tend to always accept a solution that seems best in shortsight. We just move in the downwards direction ⬇️ (negative gradient) and not upwards⬆️
So once we reach a point which is low but not the lowest, we may end up getting stuck.
🔹Either we can just stay there and accept current state as the solution, which is not very effective
🔹We can temporarily take a step in upwards direction that can get us out of the local minimum
And obvious enough we go with the second option.
To get out of local minimum we consider a random step in a direction.
If it improves the solution we always accept it. If not then we accept it anyways with a certain probability.
So we are accepting a bad move temporarily but it would make things better in the long run
T, the temperature 🌡️ parameter determines how bad a move are we willing to accept.
The higher the temperature, the more likely we are to accept a bad solution or a step upwards and vice-versa.
It doesn't end there, we also need a schedule for decreasing the value of the temperature parameter.
Can you guess the reason❓
To make sure that when we do reach near the global minimum solution we want to be less likely to accept worse solutions
There are multiple methods we can use to schedule the temperature reduction
This is how SIMULATED ANNEALING helps Gradient Descent to be better than usual.
Hope you enjoyed this!
We will see how to overcome some more problems of Gradient Descent.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Here are the links for all the notes that I have from the Andrew NG Machine Learning Course that I made back in 2016
This was my first exposure to #MachineLearning They helped me a lot and I hope anyone who's just starting out and prefers handwritten notes can reference these 👇