Thread by @apozasker

New paper out!

scirate.com/arxiv/1910.015…

It's time to talk about #machinelearning, generative models, statistical physics and spin glasses

First, I'm very happy to say this is the first purely-@ICFOnians collaboration I'm doing (although Przemek is now in @UAM_Poznan), and I've been lucky to work side-by-side with the great @gorka_mgm, who is also a @BecarioslaCaixa

But let's go to the matter. The problem we deal with has to do with #unsupervised #learning. Given some data, we want to learn its fundamental features, in order to generate more data. This is used in #generative modelling for image denoising, colorization... etc

In particular, we focus on the so-called Boltzmann machines (en.wikipedia.org/wiki/Boltzmann…). In Boltzmann machines, or BMs, every datapoint is given an energy, and the task is to lower the energy of training datapoints and increase that of the rest of possible points in your space.

Boltzmann machines are neural networks that are very easy to study for us physicists, since they can be understood as Ising models of spins. And we know that well

In fact, we know that for a good training of a Boltzmann machine, it is important to have a good characterization of the low-energy spectrum of the associated Ising model. This is, you associate the information in a datapoint with a configuration of spins, and...

... you want to have good knowledge of all those spin configurations that have low energy. It seems like an easy problem, until you realize that for random weights in your model (this is how you typically begin training) the energy landscape of the BM looks like this

Many peaks, and many, very similar valleys. This is the spin-glass phase. Knowing which valley is the lowest one is just a nightmare.

A lot of people have been trying to find better and better sampling methods that would give better access to the low-energy states. This has led to techniques such as Contrastive Divergence, PCD, Parallel Tempering... and many more

But wouldn't it be easier if we just had a smoother surface from the very beginning, where the low-energy states were easy to find?

This is precisely what we attempt in
scirate.com/arxiv/1910.015…
arxiv.org/abs/1910.01592

We take Boltzmann machine models, and Restrict their Axons (weights) by writing them as an average over some "learnable configurations", or patterns, that will be the new parameters in our model

Whenever the number of patterns is low compared to the amount of neurons in the model, the associated Ising system is outside the spin-glass phase, and thus it is easy to find the low-energy states. In fact, the patterns themselves are low-energy states!

Since the patterns themselves are low-energy states, they can be employed for computing the negative phase of the Boltzmann machine during training. Without using any sampling at all. We call this training by Pattern-InDuced correlations.

RAPID, the combination of Restricting Axons and training via Pattern-InDuced correlations, leads to HUGE improvements in training time. We find that we successfully train models in orders of magnitude less epochs

Do you want to know more? The paper can be found in

scirate.com/arxiv/1910.015…
arxiv.org/abs/1910.01592

Plus, we uploaded some sample codes to the repo

github.com/apozas/rapid

Finally, let me thank again all the @ICFOnians team, and in particular to @gorka_mgm for being such an awesome colleague!