Thread by @chelseabfinn

Convolution is an example of structure we build into neural nets. Can we _discover_ convolutions & other symmetries from data?

Excited to introduce:
Meta-Learning Symmetries by Reparameterization
arxiv.org/abs/2007.02933

w/ @allan_zhou1 @TensorProduct @StanfordAILab
Thread👇

To think about this question, we first look at how equivariances are represented in neural nets.

They can be seen as certain weight-sharing & weight-sparsity patterns. For example, consider convolutions.
(2/8)

We reparametrize a weight matrix into a sharing matrix & underlying filter parameters

It turns out this can provably represent any equivariant structure + filter parameters, for all group-equivariant convolutions with finite groups.
(3/8)

With this reparametrization, we can now optimize for equivariance separately from the parameters!

To do so, the inner loop updates only the parameters, keeping the sharing matrix fixed. In the outer loop, we meta-learn both the sharing matrix and the initial parameters.
(4/8)

In our expts, we first ask: can we discover convolutions from translationally-equivariant data?

We generate synthetic data w/ 1D translation equivariance, and compare this approach (MSR) to MAML.

MSR matches MAML+conv performance and recovers convolutional weight sharing!
(5/8)

We then ask, can we discover something *better* that convolutions when there is less or more symmetry in the data?

With the same synthetic data set-up, we find that the answer is indeed yes, and that MSR can be applied on top of convolutions to learn stronger symmetries.
(6/8)

Finally, we also aim to learn symmetries from augmented image data, to essentially _bake data augmentation into the architecture_.

Here, we find that MSR also performs well.
(7/8)

Overall, MSR provides a framework for understanding the interplay of features & structure in meta-learning.

This was our first foray into this topic, and we’d love to hear any feedback!
(8/8)