Tweet

https://twitter.com/alfcnz/status/1379112137587630087

https://twitter.com/alfcnz/status/1382178290375520256

https://twitter.com/alfcnz/status/1388654284221337613

https://twitter.com/alfcnz/status/1392083313792364553

https://twitter.com/alfcnz/status/1395116825701232642

More from @alfcnz

Alfredo Canziani

@alfcnz

19 May

@PyTorch

The fifth episode (of five) of the energy 🔋 saga is out! 🤩

In this last episode of the energy saga we code up an AE, DAE, and VAE in @PyTorch. Then, we learn about GAN, where a cost net C is trained contrastively with samples generated by another net.

A GAN is simply a contrastive technique where a cost net C is trained to assign low energy to samples y (blue, cold 🥶, low energy) from the data set and high energy to contrastive samples ŷ (red, hot 🥵, where the “hat” points upward indicating high energy).

y comes from the data set Y.
ŷ is produced by the generating network G, which maps a random vector to the input space ŷ = G(z).
To train G we simply minimise C(G(z)).

And that's it.
No fooling around with discriminators. 🥸
It's *simply* contrastive energy learning. 😇

Read 5 tweets

Alfredo Canziani

@alfcnz

11 May

The fourth episode (of five) of the energy 🔋 saga is out! 🤩

From LV EBM to target prop(agation) to vanilla autoencoder, and then denoising, contractive, and variational autoencoders. Finally, we learn about the VAE's bubble-of-bubbles interpretation.

Edit: updating a thumbnail and adding one more.

In this episode I *really* changed the content wrt last year. Being exposed to EBMs for several semesters now made me realise how all these architectures (and more to come) are connected to each other.

@ylecun

In the companion lecture (which will soon come online), @ylecun goes over a more powerful interpretation of VAE, which I still struggle to understand. As you can imagine, another tweak to my deck will occur when I'll actually get it. (Yeah, I'm slow, yet persistent.)

Read 5 tweets

Alfredo Canziani

@alfcnz

8 Apr

https://twitter.com/alfcnz/status/1380181503112527876

— Context —

Speaking about the transformer architecture, one may incorrectly talk about an encoder-decoder architecture. But this is *clearly* not the case.
The transformer architecture is an example of encoder-predictor-decoder architecture, or a conditional language-model.

https://twitter.com/alfcnz/status/1380181503112527876

The classical definition of an encoder-decoder architecture is the autoencoder (AE). The (blue / cold / low-energy) target y is auto-encoded. (The AE slides are coming out later today.)

Now, the main difference between an AE and a language-model (LM) is that the input is delayed by one unit. This means that a predictor is necessary to estimate the hidden representation of a *future* symbol.
It's similar to a denoising AE, where there is a temporal corruption.

Read 7 tweets

Alfredo Canziani

@alfcnz

23 Nov 20

@MATLAB

#AcademicChatter

Coming from engineering, I'm a former @MATLAB user, moved to @TorchML and @LuaLang, then to @PyTorch and @ThePSF @RealPython, and now I'm exploring @WolframResearch @stephen_wolfram.
For learning, one would prefer knowledge packed frameworks and documentation.

@MATLAB

In that regard, @MATLAB and @WolframResearch are ridiculously compelling. The user manuals are just amazing, with everything organised and available at your disposal. Moreover, the language syntax is logical, much closer to math, and aligned to your mental flow.

In Mathematica I can write y = 2x (implicit multiplication), x = 6, and y will be now equal 12. y is a variable.
Or I can create a function of x with y[x_] := 2x (notice that x_ means I don't evaluate y right now). Later, I can execute y[x] and get 12, as above.

Read 7 tweets

Alfredo Canziani

@alfcnz

31 Oct 20

https://twitter.com/alfcnz/status/1319121960429887488

This week we went through the second part of my lecture on latent variable 👻 energy 🔋 based models. 🤓

We've warmed up a little the temperature 🌡, moving from the freezing 🥶 zero-temperature free energy Fₒₒ(y) (you see below spinning) to a warmer 🥰 Fᵦ(y).

https://twitter.com/alfcnz/status/1319121960429887488

Be careful with that thermostat! If it's gonna get too hot 🥵 you'll end up killing ☠️ your latents 👻 and end up with averaging them all out, indiscriminately, ending up with plain boring MSE (fig 1.3)! 🤒
From fig 2.1–3, you can see how more z's contribute to Fᵦ(y).

This is nice, 'cos during training (fig 3.3, bottom) *The Force* will be strong with a wider region of your manifold, and no longer with the single Jedi. This in turns will lead to a more even pull and will avoid overfitting (fig 3.3, top). Still, we're fine here because z ∈ ℝ.

Read 6 tweets

Alfredo Canziani

@alfcnz

21 Oct 20

This week we've learnt how to perform inference with a latent variable 👻 energy 🔋 based model. 🤓
These models are very convenient when we cannot use a standard feed-forward net that maps vector to vector, and allow us to learn one-to-many and many-to-one relationships.

Take the example of the horn 📯 (this time I drew it correctly, i.e. points do not lie on a grid 𐄳). Given an x there are multiple correct y's, actually, there is a whole ellipse (∞ nb of points) that's associated with it!
Or, forget the x, even considering y alone…

there are (often) two values of y₂ per a given y₁! Use MSE and you'll get a point in the middle… which is WRONG.

What's a “latent variable” you may ask now.
Well, it's a ghost 👻 variable. It was indeed used to generate the data (θ) but we don't have access to (z).

Read 7 tweets

Share this page!

Alfredo Canziani

Try unrolling a thread yourself!

More from @alfcnz

Alfredo Canziani

Alfredo Canziani

Alfredo Canziani

Alfredo Canziani

Alfredo Canziani

Alfredo Canziani

Did Thread Reader help you today?

Like this author's thread?