Incredible writeup! Some notable 💎s:

Deepseek reduced attention complexity from quadratic to ~linear through warm-starting (w/ separate init + opt dynamics) and adapting the change over ~1T tokens.

They also use separate attention modes for disaggregated prefill vs decode (is this the first public account of arch difference between the two? 👀).

1/🧵

https://x.com/suchenzang/status/1995452641430651132

They also make several innovations to stabilize RL training (far beyond what that other "open bell labs" place published in blog posts 👀):

1) unbiased kl estimate, with different kl reg for different domains (!)

2) mask significantly negative adv sequences (to not "throw off" the model)

3) fix real-world training/inference mismatch issues with MoEs between different frameworks (preserve expert routing + preserve top-p sampling masks)

2/🧵

There seems to be a high correlation between folks who think "open source LLMs will win" & folks who

1) haven't developed any large-scale infra "in the cloud" themselves

or

2) have never priced out the cost of scaling-out "in the cloud" (for the most powerful models)

1/6 or

3) underestimate the cost of investing in data infrastructure and tooling

(have you seen why Databricks exists?)

2/6

Piling on to the pile-on (sorry - it's always easy to criticize 😛), here's a rant about benchmarks for LLMs that are used to back claims of "stronger" or "better" models.

Let's start with a tour through GPT-3's Appendix G... 1/8

https://twitter.com/drjwrae/status/1617033514037411847

First up: BoolQ. If you download the actual benchmark, it's true/false completions. GPT-3 swaps in yes/no instead. Why? Well when we did the same swap to yes/no, we saw a +10% accuracy jump on this benchmark.

Wonderful. Clearly on track for a better model already. 2/8

After ignoring the details in all these "lets-fit-a-cloud-of-points-to-a-single-line" papers (all likely wrong when you really extrapolate), @stephenroller finally convinced me to work through the math in the Chinchilla paper and as expected, this was a doozy. [1/7]

https://twitter.com/stephenroller/status/1616605686141435905

First thing to make me eye-roll a bit was this fancy equation (4) that seems to re-parameterize the key exponent terms (a,b) into (alpha,beta) to define a coefficient term G. Why this level of indirection just to define a scalar-coefficient? No idea. [2/7]