🎥 Improving language models by retrieving from trillions of tokens by @DeepMind

Paper explanation by @janithcwanni

The paper introduced Retrieval Enhanced Transformer (RETRO) - 25X smaller than GPT-3 with comparable performance. Link to paper and other related resources such as code, discussions and tweets:

📎 papers.labml.ai/paper/324a7d2e…

🎥 Lists of papers covered by popular YouTube channels

papers.labml.ai/lists

Here are some lists/channels we picked (1/9)

🧵👇 2/9

Yannic Kilcher @ykilcher

📑 papers.labml.ai/lists/yannic_k…

https://twitter.com/labmlai/status/1406203374358261760

👇

🥳 Excited to share our Chrome browser extension for papers.labml.ai 🎉

chrome.google.com/webstore/detai…

It identifies research papers mentioned in websites you visit and shows a 2-line summary, availability code/videos/discussions, popularity on Twitter, and conferences.

🧵👇 2/4 We released the source code of the extension

github.com/labmlai/chrome…

👇

If you like our highlighted papers you will love this!

We found a few awesome Github repos with highlighted/annotated research paper PDFs. We started linking them from papers.labml.ai

Here's the list of of repos and 😍 interesting papers they had:

🧵👇 2/13 Github: github.com/AakashKumarNai…
by @A_K_Nain
📎 20 ✨ 1,986 🏃 active

He has covered a wide area of papers with a lot of self-supervised papers.

👇

Patches Are All You Need? by ❓

@PyTorch Paper implementation with side-by-side notes.

📝 Annotated code nn.labml.ai/conv_mixer/ind…
📎 Paper papers.labml.ai/paper/dd638a44…

The paper introduces ConvMixer which mix patch embeddings with depth-wise and point-wise convolutions.

🧵👇 2/ The implementation is very simple and the paper presents a 280 character version of the @PyTorch model code - fits a tweet 💪

Our implementation is a bit lengthy (hopefully easier to understand 😁)

👇

Annotated @PyTorch implementation of "Denoising Diffusion Probabilistic Models" by @hojonathanho @ajayj_ @pabbeel @berkeley_ai

📝 Annotated code nn.labml.ai/diffusion/ddpm…
🖥 Github github.com/labmlai/annota…
📎 Paper papers.labml.ai/paper/2006.112…

🧵👇 2/ This removes noise (denoise) step-by-step to generate images. It adds noise to an image from the dataset iteratively and a model is trained to predict the noise at each step.

👇

Annotated @PyTorch implementation of Primer EZ

📝 Code/notes nn.labml.ai/transformers/p…

📄 Paper papers.labml.ai/paper/2109.086…

Primer is a transformer model found using an evolutionary search on architectures. Primer EZ has the two most robust findings.

🧵👇 2/4) The paper notes that achieves the performance of T5 on C4 LM dataset with 4.2X less compute.

We experimented with a small dataset on a single GPU🥺 and it trains much faster than the vanilla transformer.

app.labml.ai/run/30adb7aa1a…

👇

Learning to play Kuhn Poker with Monte Carlo Counterfactual Regret Minimization (MC-CFR) in #python

📝 Code/Tutorial: nn.labml.ai/cfr/index.html

This isn't deep learning. But it'll be interesting if you do machine learning, like incomplete information games or play #poker.

🧵👇 2/8) Kuhn Poker is a simple 2-player betting game with three cards (A, K, Q). A single card is dealt to each player. Players take turns betting chips and the player with the higher card wins the chips. If a player folds the other player wins the chips.

👇

1/4) Simple StyleGAN2 implementation (with notes) in @PyTorch in less than 500 lines of code (including training)

Annotated Code: nn.labml.ai/gan/stylegan/i…

Github: github.com/lab-ml/annotat…

Paper: arxiv.org/abs/1912.04958

@jaakkolehtinen @NVIDIAAI

🧵👇 2/4) StyleGAN2 is based on StyleGAN which is based on ProgressiveGAN (by the same authors). We have briefly discussed these models.

StyleGAN paper: arxiv.org/abs/1812.04948
ProgressiveGAN paper: arxiv.org/abs/1710.10196

👇