🥳 NEW LECTURE 🥳
“Set to set” and “set to vector” mappings using self/cross hard/soft attention. We combined a (two) attention module(s) with a (two) k=1 1D convolution to get a transformer encoder (decoder).
Slides: github.com/Atcold/pytorch…
Notebook: github.com/Atcold/pytorch…
“Set to set” and “set to vector” mappings using self/cross hard/soft attention. We combined a (two) attention module(s) with a (two) k=1 1D convolution to get a transformer encoder (decoder).
Slides: github.com/Atcold/pytorch…
Notebook: github.com/Atcold/pytorch…
This week's slides were quite dense, but we've been building up momentum since the beginning of class, 3 months ago.
We recalled concepts from:
• Linear Algebra (Ax as lin. comb. of A's columns weighted by x's components, or scalar products or A's rows against x)
We recalled concepts from:
• Linear Algebra (Ax as lin. comb. of A's columns weighted by x's components, or scalar products or A's rows against x)
• Recurrent Nets (stacking x[t] with h[t–1] and concatenating W_x and W_h)
• Autoencoders (encoder-decoder architecture)
• k=1 1D convolutions (that does not assume correlation between neighbouring features and act as a dim. adapter)
and put in practice with @PyTorch.
• Autoencoders (encoder-decoder architecture)
• k=1 1D convolutions (that does not assume correlation between neighbouring features and act as a dim. adapter)
and put in practice with @PyTorch.
Notice how you can smoothly go from hard to soft attention by switching between argmax and softargmax (which most of you still call “softmax”). Hard attention implies a one-hot vector a, while soft attention gives your pseudo probabilities.
Once again, this architecture deals with *sets* of symbols!
There is no order. Therefore, computations can be massively parallelised (they are just a bunch of matrix products, afterall).
Just be aware of that t × t A matrix that could blow up, if t (your set length) is large.
There is no order. Therefore, computations can be massively parallelised (they are just a bunch of matrix products, afterall).
Just be aware of that t × t A matrix that could blow up, if t (your set length) is large.
Just a final recap, there is *one* and *only one* _query_ (I'd like to cook a lasagna) that I'm going to check against *all* _keys_ (recipes titles) in order to retrieve *one* (if hard) or *a mixed* (if soft) _value_ (recipe to prepare my dinner with).
Me, hungry, during class = decoder.
My granny, knowing all recipes names (keys) and preparations (values) = encoder.
Me, figuring out what I want = self-attention.
Me, asking granny = cross-attention.
Dinner = yay!
My granny, knowing all recipes names (keys) and preparations (values) = encoder.
Me, figuring out what I want = self-attention.
Me, asking granny = cross-attention.
Dinner = yay!
I'm done.
Next week: Graph Neural Nets (if it's taking me less than a week to learn about them).
Next week: Graph Neural Nets (if it's taking me less than a week to learn about them).
