“Carmack describes how this is like a game—as soon as you get an idea you try to defeat it. You'll be able to generate more ideas because you freed up mental space. Moreover, your existing ideas will even be stronger because they survived heavy scrutiny.”
amasad.me/carmack
amasad.me/carmack
This only works if you have a good intuition about whether things will work in the first place, which may not be the case for neural networks.
At the end of the day, it’s hard to beat quickly coding up, proof-of-concept notebooks on toy tasks to quickly test out new ideas.
At the end of the day, it’s hard to beat quickly coding up, proof-of-concept notebooks on toy tasks to quickly test out new ideas.
When Adam Gaier first tried to test weight agnostic neural networks using random weights, we didn’t get any good results. He then made a bug in the code that accidentally forced all the weights to be shared, and suddenly we got much better results, and based our paper on the bug.
