Delighted to share our work on reasoning-modulated representations! Contributed talk at @icmlconf SSL Workshop 🎉

Algo reasoning can help representation learning! See thread👇🧵

w/ Matko @thomaskipf @AlexLerchner @RaiaHadsell @rpascanu @BlundellCharles
We study a very common representation learning setting where we know *something* about our task's generative process. e.g. agents must obey some laws of physics, or a video game console manipulates certain RAM slots. However...
...explicitly making use of this information is often quite tricky, every step of the way! Depending on the circumstances, it may require hard disentanglement of generative factors, a punishing bottleneck through the algorithm, or necessitate a differentiable renderer!
Algorithmic reasoning blueprint to the rescue!

In RMR, we show that we can encapsulate the x_bar -> y_bar path using a high-dimensional GNN, pre-trained on large quantities of data (which we can usually pre-generate, even synthetically).

This alleviates all of the above issues.
We recover significant improvements over a baseline without pre-training. N.B. RMR still needs to learn how to meaningfully use representations from a completely different task!

Our evaluation spans bouncing balls data (elastic collisions) & Atari trajectories (RAM transitions).
Lastly, alongside other recent works like XLVIN, we believe this is only one of many exciting uses of algorithmic reasoning to come in the near future! Watch this space 🎆

Any thoughts, comments and feedback is highly welcome! :)

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Petar Veličković

Petar Veličković Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @PetarV_93

21 Jul
We release the full technical report & code for our OGB-LSC entry, in advance of our KDD Cup presentations! 🎉

See thread 🧵 for our insights gathered while deploying large-scale GNNs!

with @PeterWBattaglia @davidmbudden @andreeadeac22 @SibonLi et al. Image
For large-scale transductive node classification (MAG240M), we found it beneficial to treat subsampled patches bidirectionally, and go deeper than their diameter. Further, self-supervised learning becomes important at this scale. BGRL allowed training 10x longer w/o overfitting.
For large-scale quantum chemical computations (PCQM4M), going deeper (32-50 GNN layers) yields monotonic and consistent gains in performance. To recover such gains, careful regularisation is required (we used Noisy Nodes). RDKit conformers provided a slight but significant boost.
Read 4 tweets
5 Jul
I firmly believe in giving back to the community I came from, as well as paying forward and making (geometric) deep learning more inclusive to underrepresented communities in general.

Accordingly, this summer you can (virtually) find me on several summer schools! A thread (1/9)
At @EEMLcommunity 2021, I will give a lecture on graph neural networks from the ground up, followed by a GNN lab session led by @ni_jovanovic. I will also host a mentorship session with several aspiring mentees!

Based on 2020, I anticipate a recording will be available! (2/9)
Alongside @mmbronstein @joanbruna @TacoCohen, I will be co-hosting a course on Geometric Deep Learning for the African Master of Machine Intelligence @AIMS_Next.

This will closely follow our recently-released proto-book and we hope to make materials more broadly available. (3/9)
Read 9 tweets
28 Apr
Proud to share our 150-page "proto-book" with @mmbronstein @joanbruna @TacoCohen on geometric DL! Through the lens of symmetries and invariances, we attempt to distill "all you need to build the architectures that are all you need".

More info below! 🧵
We have investigated the essence of popular deep learning architectures (CNNs, GNNs, Transformers, LSTMs) and realised that, assuming a proper set of symmetries we would like to stay resistant to, they can all be expressed using a common geometric blueprint.

But there's more!
Going further, we use our blueprint on less standard domains (such as homogeneous groups and manifolds), showing that the blueprint allows for nicely expressing recent advances in those areas, such as Spherical CNNs, SO(3)-Transformers, and Gauge-Equivariant Mesh CNNs.
Read 5 tweets
24 Apr
The crowd has spoken! 🙃 A thread with early-stage machine learning research advice follows below. 👇🧵

Important disclaimer before proceeding: these are my personal views only, and likely strongly biased by my experiences and temperament. Hopefully useful nonetheless! 1/15
During the early stages of my PhD, one problem would often arise: I would come up with ideas that simply weren't the right kind of idea for the kind of hardware/software/expertise setup I had in my department. 2/15
This would lead me on 'witch hunts' that took months (sometimes forcing me to spend my own salary on compute!). Game-changer for me was corresponding w/ researchers that are influential to the work I'd like to do: first learn from their perspectives, eventually internships. 3/15
Read 15 tweets
16 Nov 20
Over the past weeks, several people have reached out to me for comment on "Combining Label Propagation and Simple Models Out-performs Graph Neural Networks" -- a very cool LabelProp-based baseline for graph representation learning. Here's a thread 👇 1/14
Firstly, I'd like to note that, in my opinion, this is a very strong and important work for representation learning on graphs. It provides us with so many lightweight baselines that often perform amazingly well -- on that, I strongly congratulate the authors! 2/14
I think most of the discussion comes from the title -- most people reaching out to me ask "Does this mean we don't need GNNs at all?", "Have GNNs been buried?", etc.

In reality, this work reinforces something we've known in graph representation learning for quite some time. 3/14
Read 14 tweets
17 Sep 20
As requested , here are a few non-exhaustive resources I'd recommend for getting started with Graph Neural Nets (GNNs), depending on what flavour of learning suits you best.

Covering blogs, talks, deep-dives, feeds, data, repositories, books and university courses! A thread 👇
For blogs, I'd recommend:
- @thomaskipf's post on Graph Convolutional Networks:…
- My blog on Graph Attention Networks:
- A series of comprehensive deep-dives from @mmbronstein:…
For a comprehensive overview of the area in the form of a talk, I would highly recommend @xbresson's guest lecture at NYU's Deep Learning course:

Read 8 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!