My Authors
Read all threads
🥳 NEW LECTURE 🥳
Graph Convolutional Networks… from attention!
In attention đť’‚ is computed with a [soft]argmax over scores. In GCNs đť’‚ is simply given, and it's called "adjacency vector".
Slides: github.com/Atcold/pytorch…
Notebook: github.com/Atcold/pytorch…
Summary of today's class.

Slide 1: *shows title*.
Slide 2: *recalls self-attention*.
Slide 3: *shows đť’‚, points out it's given*.
The end.

Literally!
I've spent the last week reading everything about these GCNs and… LOL, they quickly found a spot in my mind, next to attention!
The key concept here is the *sparsity* (constraints) of the graph.
In self-attention, every element in the set looks at each and every other element.
If a sparse graph is given, we limit each element (node / vertex) to look only at a few other elements (nodes / vertices).
Now, if every incoming node is treated equally, this might seem a little weak, as learning algorithm.
Don't you worry! Let's reason on the edges (connecting the incoming nodes).
Enters the "gated" GCN, where the incoming node / message is modulated by a gate đťś‚.
Here đťś‚ if function of the representation (embedding / feature) of the incoming edge, which is a normalised sigmoid MLP (k=1 1D CNN, actually).
Finally, because-why-not, let's add some residual connections to smooth a little the loss landscape.

Tadaaaa!
The end.
P.S. I can't remember the last time I studied this much in such a short lapse of time. My brain is trying to escape my skull… I feel drunk and nauseated. But yes, one more lesson added to my collection.

Next week (if I survive ICLR): training your own energy model from scratch.
Missing some Tweet in this thread? You can try to force a refresh.

Enjoying this thread?

Keep Current with Alfredo Canziani

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!