Learning rate is one of the most important parameter in Machine Learning Algorithms.📈
You must have seen learning rates something like 0.01, 0.001, 0.0001....
In other words, always in the logarithmic scale. Why?
You must have seen learning rates something like 0.01, 0.001, 0.0001....
In other words, always in the logarithmic scale. Why?
What happens if we just take random values between 0 and 1?
If we take random values between 0 and 1, we would have a probability of only 10% to get the values between 0 an 0.1, rest 90% of the values would be between 0.1 and 1.
But why do we want between 0 and 0.1?
But why do we want between 0 and 0.1?
Because the preference is to find the correct scale not necessarily the number, optionally we can later perform a fine-grained search in the suitable range.
We want to test all orders of magnitude which can be done using a log-scale. So we define our search space somewhat like:
We want to test all orders of magnitude which can be done using a log-scale. So we define our search space somewhat like:
Using log-scale is a common practice for selecting hyper-parameters but it usually depends on the type of parameter.
One common exception which uses linear scale is Dropout!
One common exception which uses linear scale is Dropout!
• • •
Missing some Tweet in this thread? You can try to
force a refresh
