As requested , here are a few non-exhaustive resources I'd recommend for getting started with Graph Neural Nets (GNNs), depending on what flavour of learning suits you best.
Covering blogs, talks, deep-dives, feeds, data, repositories, books and university courses! A thread 👇
A clear step towards achieving my dream: building AI that assists competitive programmers 🧑💻
“This is an exciting approach to combine work of human competitive programmers and LLMs, to achieve results that neither would achieve on their own.” --Petr Mitrichev
Details below! 🧵
There's been a rightful surge of AI-powered competitive programming systems, typically deployed on classical contests such as Codeforces.
While very impressive results have been achieved (ELO ~1,900), they are still significantly away from the highest percentiles of competitors.
In contrast, combinatorial tasks present a very different challenge: tasks are intractable (NP-hard).
This levels the playing field: humans will not know the optimal solution, and suboptimal solutions score >0 points. This allows AI to explore in a way that complements humans 🔎
Read on to see how we expose fundamental weaknesses of decoder-only Transformers on important tasks (e.g. copying & counting) + simple ways to make things a bit easier on the Transformer :)
Work led by @fedzbar for his @GoogleDeepMind placement!
We start by asking a frontier LLM a simple query: copy the first & last token of bitstrings.
Not only does it fail past a certain length, it also fails in a very specific way: it fails when there's repetition (111...10), and it fails to copy the _last_ token, never the first.
This leads to our first result -- representational collapse.
We prove there must exist pairs of different inputs for which their last token representations cannot be distinguished.
To prove this, we use bitstrings of the form 11...10, where repetitions exacerbate the problem.
If you are @LogConference, come to the virtual Poster Session in ~20 minutes -- we have _four_ posters on algorithmic alignment, reasoning and over-squashing in GNNs! 🕸️🍾🌐 Several of them are award-winning!
You're welcome to stop by for a chat. 😊
See the 🧵for details... 🔢
🌐 In "Reasoning-Modulated Representations", Matko Bošnjak, @thomaskipf, @AlexLerchner, @RaiaHadsell, Razvan Pascanu, @BlundellCharles and I demonstrate how to leverage arbitrary algorithmic priors for self-supervised learning. It even transfers _across_ different Atari games!
🤖 In "Continuous Neural Algorithmic Planners", @heyu0208, @pl219_Cambridge, @andreeadeac22 and I show how the ideas from XLVIN paper can generalise to continuous-action-space environments (such as MuJoCo!). CNAP won the Best Paper Runner-up Award at GroundedML @ ICLR'22!
We made careful modifications to our content, making it more streamlined & accessible!
Featuring a revamped introductory lecture, clearer discussion of Transformers & a new lecture going beyond groups, into the realm of category theory! 🐲
Algorithmic reasoning has emerged as a very important area of representation learning! Many key works (feat. @KeyuluXu@jingling_li@StefanieJegelka@beabevi_@brunofmr) explored important theoretical and empirical aspects of algorithmic alignment.
Critically, each one of these works (incl. mine!) operates over its own datasets, often making it hard to directly compare insight among papers.
Further, generating adequate datasets requires knowledge of theoretical computer science, raising barrier of entry to the field.
It required no (apparent) novel research (though it could enable lots of new research!), I had the theoretical skills to understand everything that needs to be implemented, and it amounted to standard supervised learning! 2/11
So I started implementing by myself. What could possibly go wrong? Turns out, pretty much everything. :)
Indeed, I understood all I needed to write generators of the data. But this didn't mean I knew how to most efficiently extract it, organise it, and make it accessible! 3/11