,
5 tweets,
3 min read
Read on Twitter
Weight Agnostic Neural Networks 🦎
Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.
Article: weightagnostic.github.io
PDF: arxiv.org/abs/1906.04358
Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.
Article: weightagnostic.github.io
PDF: arxiv.org/abs/1906.04358
Networks that already work with random weights are not only easily trainable, they also offer other benefits. We can give the same network an ensemble of different random weights to boost performance, without any training.
An MNIST classifier evolved to work with random weights:
An MNIST classifier evolved to work with random weights:
Key idea is to search for architectures by de-emphasizing weights. During the search, networks are assigned a single shared weight value at each rollout, and optimized to perform well over a wide range of weight values. As a bonus, we get to bypass the costly inner training loop!
This work was led by Adam Gaier, who did a 3-month internship in the Google Brain team in Tokyo. This idea came out after a few drinks in Roppongi. He has done some fantastic work in the Neuroevolution area in the past and is active in the GECCO community. bit.ly/2Kbx0EZ
Some failure cases look interesting. A WANN evolved to walk ahead sometimes perform non-trivial actions like balancing when using bad weights.
