Thread by @Smerity

Deep learning training tip that I realized I do but never learned from anyone - when tweaking your model for improving gradient flow / speed to converge, keep the exact same random seed (hyperparameters and weight initializations) and only modify the model interactions.

- Your model runs will have the exact same perplexity spikes (hits confusing data at the same time)
- You can compare timestamp / batch results in early training as a pseudo-estimate of convergence
- Improved gradient flow visibly helps the same init do better

Important to change out the random seed occasionally when you think you've isolated progress but minimizing noise during experimentation is OP. You're already dealing with millions of parameters and billions of calculations. You don't need any more confusion in the process.

Anyone else doing this? As noted I never learned it explicitly, just a habit I got into. I tend to think most BigCo will be running hyperparameter sweeps / randomization due to the environment but maybe others are doing this on their own?

This may or may not be a soft pseudo-science variant of the Lottery Ticket Hypothesis / @hardmaru et al's Weight Agnostic Neural Networks. Either way it has worked multiple times over multiple datasets for me and the results seem to generalize.