Thread by @ai__pub

// SCALING LAWS, Explained (Part I) //

You've heard "Scale is All You Need", seen graphs with straight lines going down, others muttering that "AGI is coming".

What exactly are scaling laws? What is being scaled? And why is everyone excited (or scared) about them?

1/18

First, a summary in two tweets:

Scaling laws show that large language models' (LLMs') loss decreases when you increase the model's:
- # parameters,
- training data,
- training compute budget

In a mathematically predictable way.

2/18

Why does this matter?

It means we don't need to make theoretical or architectural advances to significantly power up LLMs.

Just let the GPUs rip, baby!

(There's a second, much more important reason at the end of the thread!)

3/18

Now, the details. First, the form of the law:

With L the test loss,

x = one of compute, dataset size, or model parameters, and *not* bottlenecked by the other two, empirically we find

L = C/x^a

For constants C and a that depend on the scaling type (see below!).

4/18

Alternatively we have log L ~ -log(a)*log(x), so a linear relationship on a log-log scale.

(Hence the straight-line log-log charts.)

5/18

What's the loss L, you ask?

It's the "cross-entropy", which in this case is also equal to the (average) log-likelihood. Formula below!

Intuitively, it's the (scaled, -log) chance that your LLM would actually generate your text dataset. You want this chance to be high.

6/18

Now that you get the basics, here's a puzzle:

The scaling law exponents are between 0.05 and 0.1. These exponents are tiny.

Concretely, if you 1,000x'd your training data, your loss would only go down by a factor of 1/(1,000)^0.1 ~ 50%!

So why care about scaling laws?

7/18

Here comes the kicker - why you should *actually* care about scaling laws, why some say "Scale is All You Need", why others say "AGI is coming":

*Minor* improvements in test loss give *massive* generalization and new capabilities for LLMs.

An example from DeepMind:

8/18

DeepMind trained two LLMs, Gopher and Chinchilla.

Chinchilla was trained more efficiently resulting in a cross-entropy of 1.97, vs. Gopher's 2.05 - only a 4% improvement.

Yet Chinchilla blew Gopher out of the water on a range of high-level tasks like high school math.

9/18

Specifically, Chinchilla performed on average 10x better than Gopher on MMLU, a multitask language understanding dataset ranging from exams in professional medicine to high school chemistry.

Chinchilla beat Gopher by over 30x in college physics, with only 4% lower loss!

10/18

This is *real* reason to care about scaling laws, why some say that "AGI is coming". The reasoning chain is:

1. Scale up data, params, GPUs go brrr -->
2. *Minor* yet predictable decrease in model loss -->
3. *Massive* increase in LLM capabilities.

Be afraid! (perhaps)

11/18

That's a wrap on scaling laws! If you read the thread carefully, you now understand:

1. The mathematical form of scaling laws
2. The cross-entropy loss
3. That scaling laws lead to minor improvements in loss...
4. But massive improvements in LLM capabilities

12/18

This thread was an introduction to scaling laws, and largely a walk-through of OpenAI's 2020 paper that discovered them.

Later this week we'll do Part II on the limits of scaling laws, scaling laws and data, and the 2022 Chinchilla paper!

arxiv.org/abs/2001.08361

13/18

To buy the shirt from this thread, made by @ethanCaballero, go here: agiwear.ai

(Not a paid promotion, lol)

14/18

For those new to the channel, a bit about us:

We now do 3 things (with more cooking in the kitchen...):

1) We publish explainer threads like the one you just read. Here's another on Stable Diffusion that blew up last week:

https://twitter.com/ai__pub/status/1561362542487695360

15/18

2) We publish a weekly "Best of AI Twitter" thread - here's last week's:

https://twitter.com/ai__pub/status/1562818298311389184

16/18

3) We're building a job board for ML engineers and SWEs who want to work on ML products - out in a week!

We're also building on a solution to match the above talent with companies building cutting-edge ML products, but that'll take a month or two.

https://twitter.com/ai__pub/status/1562818359858569229

17/18

That's a wrap! Check in later this week for the job board and Part II on scaling laws.

Comments and DMs are always open if there's an explainer thread topic you'd like covered!

Oh, and tomorrow I'll post a thread of resources to learn more about scaling laws.

18/18

To learn more about scaling laws, check out these resources:

https://twitter.com/ai__pub/status/1565047584057462784

19/18