Post

https://x.com/scaling01/status/2067954502435480050

More from @scaling01

Lisan al Gaib

@scaling01

Jun 7

https://twitter.com/chamath/status/2063292917964517830

the "narrow capability gap" in question

let's put this to rest please
I can't hear the coping anymore

https://twitter.com/chamath/status/2063292917964517830

Read 9 tweets

Lisan al Gaib

@scaling01

Mar 3

The big China update for LisanBench!

Let's make this sweet and short, with some emojis, so you know which scores made me happy.

# 9 Step 3.5 Flash !!
# 14 Kimi-K2.5 Thinking :/
# 18 Seed 2.0 Pro :)
# 24 Seed 1.8 :)
# 25 GLM-5 :(
# 26 Seed 2.0 Lite :)
# 27 Minimax M2.5 :/
# 32 Seed 2.0 Mini :)

not serious results - have to rerun although the 397B results looked fine:
# 21 Qwen3.5 397B A17B :)
# 40 Qwen3.5 35B A3B :/ (few buggy responses)
# 44 Qwen3.5 122B A10B :( (lots of buggy responses)

(all with thinking, and tested with official providers. checked all the responses for errors!)

- Step 3.5 with a very impressive score. It uses an insane amount of tokens, but the validity ratio and the score honestly speaks for itself. It makes use of these tokens. Similar to DeepSeek-V3.2 Speciale. Look at the far right of the plot.

- Kimi-K2.5 has basically the same score as K2.
I can't say exactly how much more efficient it is, because I couldn't get K2 Thinking to run again. But I remember K2 taking around 40k token for each response. So we might be talking about 2x more efficient reasoning!

- Seed 2.0 Pro was impressive especially in terms of reasoning efficiency

- Qwen3.5 397B A17B outperforms GLM-5 and Minimax M2.5 which is good, but also uses more tokens.

- Qwen3.5 35B A3B is okay, still below GPT-OSS-20B. Had a couple of responses where it included the reasoning, which makes the score 0 for that word.

- Qwen3.5 122B A10B was bugged. It was returning the reasoning for like half of all responses. Should score much higher without buggy responses.

- GLM-5 was somewhat disappointing. I thought it would get on a level with Kimi-K2.5. But I checked all responses manually, no faulty responses like the smaller Qwen3.5 models.

- MiniMax M2.5 score is fine, but nothing crazy. GPT-OSS-120B kinda mogs it. Minimax also had issues with the output format. It responded with the list of words, but after like 100 words it would go back to reasoning or changing the format.

- Seed 1.8, 2.0 Lite and Mini perform all well for their price and are all super token efficient.

Usually all models should start their raw response with the starting word that was given to them. Like everything below 97% on this chart is kinda sus.

So Qwen3.5 122B A10B was clearly bugged and it seems like Llama3.1 8B was too (not gonna rerun llama tho)

I had to run Kimi-K2.5 twice, because last time I tested LisanBench with Qwen3 I had to add a /nothink tag, that I then forgot.

It didn't really do much, scores of 1893 vs 1849. This is also a nice demonstration of how accurate the scores are.

This is currently the top 50:

--- LisanBench Ranking ---
# 1 Opus 4.5 Thinking 5393.00
# 2 DeepSeek V3.2 Speciale 4711.33
# 3 Gemini 3 Pro Preview 4661.33
# 4 Grok 4 3885.17
# 5 GPT-5.2 3097.33
# 6 o3 2977.33
# 7 Grok-4-Fast 2664.20
# 8 Sonnet 4.5 Thinking 2578.33
# 9 Step 3.5 Flash 2411.33
# 10 DeepSeek V3.2 Thinking 2327.00
# 11 Sonnet 4 Thinking 2213.33
# 12 GPT-5 2116.33
# 13 Gemini 3 Flash Preview 2091.33
# 14 Kimi-K2.5 Thinking 1893.67
# 15 Kimi-K2 Thinking 1860.00
# 16 Kimi K2.5 v2 Thinking 1849.67
# 17 Grok-4.1-Fast Thinking 1806.00
# 18 Doubao Seed 2.0 Pro 1740.00
# 19 GPT-5-Mini 1546.33
# 20 o3 Mini 1375.67
# 21 Qwen3.5 397B A17B 1235.00
# 22 GPT-5-Nano 1172.00
# 23 GPT-OSS-120B 1109.00
# 24 Doubao Seed 1.8 1106.67
# 25 GLM-5 1054.00
# 26 Doubao Seed 2.0 Lite 992.33
# 27 Minimax M2.5 867.50
# 28 Grok-3-Mini 865.00
# 29 Qwen3 235B A22B 847.33
# 30 o4 Mini 791.00
# 31 Grok-3 748.67
# 32 Doubao Seed 2.0 Mini 665.67
# 33 Opus 4 589.33
# 34 GPT-OSS-20B 571.33
# 35 Gemini 2.5 Pro 566.33
# 36 Sonnet 3.7 512.00
# 37 Gemini 1.5 Pro 497.67
# 38 Sonnet 4 469.00
# 39 Sonnet 3.5 461.33
# 40 Qwen3.5 35B A3B 427.33
# 41 GLM 4.5 416.67
# 42 DeepSeek-V3 411.00
# 43 Opus 4.1 405.00
# 44 Qwen3.5 122B A10B 395.67
# 45 DeepSeek V3.2 392.33
# 46 Gemini 2.5 Flash Thinking 372.67
# 47 Sonnet 3.5 (old) 366.00
# 48 Olmo 3 32B 359.00
# 49 Kimi-K2 339.00
# 50 GPT-4o 317.33

Read 7 tweets

Lisan al Gaib

@scaling01

Feb 20

x.com/i/article/2024…

gonna create a lil fork of the arc agi 3 repo so you can try the agent or maul me if I fucked something up (again)

link to my github is on my acc, will push it sometime in the next 24 hours

Read 6 tweets

Lisan al Gaib

@scaling01

Jan 6

https://twitter.com/scaling01/status/2008387917899546671

GPT-5.2 just wasn't a very good product

just thinking too long, too slow and too expensive with the random unjustified price hike

the price hike is a good indication of where it's going for OpenAI, managers have taken over

it's no longer scientists in charge

https://twitter.com/scaling01/status/2008387917899546671

again, not saying it's not smart

I've warned multiple times of exactly this happening but whatever

there won't be a second ChatGPT rebellion

I gave them a chance

now that they have a large user case enjoy the coming price hikes / limit decreases on your subscription plans

Read 5 tweets

Lisan al Gaib

@scaling01

Jan 1

My predictions for 2026:

Coding and Mathematics AGI
- METR 50% time horizons above 24 hours - my mean estimate is 30.8 hours, 2 day time horizons possible within frontier labs when accounting for 60 day lag
- if 2025 was the year of agents, then 2026 will be the year of multi-agent systems
- agents delegating work to subagents -> the start of the agent economy and the great unhobbling!

Most of our current math and coding benchmarks will get saturated!
- Epoch Capabilities Index ( > 175 )
- FrontierMath Levels 1-3 ( > 95% )
- ARC-AGI 1 and 2 ( > 95% )
- SimpleQA verified ( > 95% )
- Simple-Bench ( > 90% )
- SWE-Bench-verified ( > 90% )
- Terminal-Bench 2 ( > 90% )
- WeirdML v2 ( > 85% )
- Humanities Last Exam ( > 80% )
- FrontierMath Level 4 ( > 75% )
- Cybench ( > 70% )
- GDPval ( > 70 % win rate, no ties)
- GSO ( > 65% )
- ARC-AGI-3 ( > 60% and > 80% if they go for o3-preview comparable compute budgets or continual learning breakthrough happens)

- more evals like gdpval that capture economic value of models and systems
- big focus white collar work and large acceleration of science: specifically i see acceleration in medicine, biology, chemistry, finance, legal, administrative work
- automation of white collar work will be enabled by having reliable and fast computer use agents
- reliable computer use agents will also have implications for how you use the internet. this is OpenAI's big goal: become the hub to the internet and delegate shopping and whatever to agents!

Big models launches to get hyped for in 2026:
- Claude 5 - Claude 5.5
- Gemini 3.5 - Gemini-4
- GPT-5.3 - GPT-6
(everything in between possible, but Gemini 4 ~ 80%, Claude 5.5 ~ 70%, GPT-6 ~ 60% likely before 2027)
- DeepSeek-V4
- Grok-5
- Qwen-4
- Kimi-K3, GLM-5, MiniMax M3
- more korean models and a bunch of american open-source models :)

The gap between closed and open labs will narrow in H1 2026 due to DeepSeek-V4, then widen in the later half of the year, especially on economically valuable tasks.
Closed models will be much more reliable. But we will still have Opus 4.5+ level open models by the end of 2026.

Most frontier models will be around 5-10T params. If we see GPT-6 and Gemini-4 at the end of 2026 10T+ param models are possible. These models + harnesses will be the first not research agents. We should also see much better live models with voice and video mode.

Model architecture:
- we will see both, more efficient architectures and more expressive architectures!
- hybrid architectures for even longer context windows, diffusion models for speed on edge devices, but also models that double down on full attention or even more expressive attention mechanisms
- looped language models, other recurrent architectures and continual learning will enable much smaller reasoning models! (TRM on ARC-AGI has paved the way for the reasoning core)
- big improvements in reasoning efficiency

in my 2025 prediction I included a prediction for 2026 that I stand by:
- "someone (Anthropic) figures out efficient test-time-training [...], this will be the next paradigm for 2026 and lead to superintelligence"

General outlook and some random thoughts:
- it will be clear to everybody that Anthropic has the mandate and is ahead of everyone else
- OpenAI, Anthropic and Google will remain frontier labs
- decent chance that Anthropic overtakes OpenAI's valuation and both are valued > 1T
- DeepSeek will join them with V4 as THE chinese frontier lab
- xAI will likely repeat Grok-4, Grok-5 will be great on benchmarks but Elon persists on slop-maxxing the model
- AI generated video content will take off with Veo-4 and Sora-3, consistent minute long videos will be possible
- embodied intelligence will start to take off by RL through world models
- full self-driving solved, waymo and tesla everywhere
- the stock market will have a 20%+ drawdown
- 15% chance of OpenAI going bankrupt and getting acquired by Microsoft due to collapse of oracle or a market crash, caused by rapidly deteriorating economic situation (unemployment, inflation)
- push against AI will become a common theme in most advanced western economies as unemployment rises
- populist right wing parties continue to gain traction in europe
- trump/republicans will lose midterm elections

On multi-agent systems
- they are great because they allow systems to work in parallel
- the main agent can avoid keeping every detail and reasoning step in context and avoids context rot
- the agent economy is an economy where AI agents pay other AI agents to do work
- i think agent harnesses /scaffolding will take off too, like claude-code but for general purpose

On continual learning:
- my original prediction included that it is achieved by using mech-interp, but the most likely candidates are:
- brute-force through smarter use of .md files, something like skills, basically self-meta-prompting: LLMs writing and optimizing instructions for itself
- nested learning architectures like HOPE
- targeted training via mech-interp, LoRa on steroids

We are approaching continual learning no matter what.
Release cycle times are decaying every year as FLOPS become more abundant.

In 2023 it was roughly every 6 months, 2024 every 3 months, now frontier labs like OpenAI, Google, Anthropic are approaching model releases monthly.

In the limit this will also lead to continual learning, even without any fancy new tech.

Read 12 tweets

Lisan al Gaib

@scaling01

Jun 8, 2025

A few more observations after replicating the Tower of Hanoi game with their exact prompts:

- You need AT LEAST 2^N - 1 moves and the output format requires 10 tokens per move + some constant stuff.
- Furthermore the output limit for Sonnet 3.7 is 128k, DeepSeek R1 64K, and o3-mini 100k tokens. This includes the reasoning tokens they use before outputting their final answer!

- all models will have 0 accuracy with more than 13 disks simply because they can not output that much!

- the max solvable sizes WITHOUT ANY ROOM FOR REASONING (floor(log2(output_limit/10)))
DeepSeek: 12 disks
Sonnet 3.7 and o3-mini: 13 disks

- If you actually look at the output of the models you will see that they don't even reason about the problem if it gets too large:
"Due to the large number of moves, I'll explain the solution approach rather than listing all 32,767 moves individually"

- At least for Sonnet it doesn't try to reason through the problem once it's above ~7 disks. It will state what the problem and the algorithm to solve it and then output its solution without even thinking about individual steps.

- it's also interesting to look at the models as having a X% chance of picking the correct token at each move
- even with a 99.99% probability the models will eventually make an error simply because of the exponentially growing problem size

Decomposing helps the model to focus more on reasoning as it keeps the problem size smaller but it will basically get lost in the algorithm and repeat steps.
It needs the history to pick up where it left, although Tower of Hanoi is in theory stateless. Like after each move the optimal move only depends on the current state.

This was with Gemini 2.0 Flash. So not a reasoning model. But as you can see decomposing the full problem that should be solvable into 2^n-1 steps, into chunks of 5 steps at a time worsens performance.

But I also observed this peak in token usage across the models I tested at around 9-11 disks.

That's simply the threshold where the models say: "Fuck off I'm not writing down 2^n_disks - 1 steps"

Read 8 tweets

Share this page!

Enter URL or ID to Unroll

Lisan al Gaib

Try unrolling a thread yourself!

More from @scaling01

Lisan al Gaib

Lisan al Gaib

Lisan al Gaib

Lisan al Gaib

Lisan al Gaib

Lisan al Gaib

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?

Send Email!