Investigating the trajectory of AI for the benefit of society.
2 subscribers
Historically, technology adoption took decades. For example, telephones took 60 years to reach 70% of US households. But tech diffuses faster and faster over time, and we should expect AI to continue this trend. 
This evaluation is not directly comparable to those on Epoch AI’s benchmarking hub, as it uses a different scaffold. First, we did not run the model ourselves—we only graded the outputs provided by OpenAI and don’t have access to their code to run the model. Second, ChatGPT agent has access to tools not available to other models we've assessed—most notably browser tools, which may have helped on questions related to recent research papers. Finally, the evaluation allowed up to 128K tokens per question, compared to our standard 100K; this difference is unlikely to have significantly affected results.
@GregHBurnham It will be tempting to focus on whether an AI system gets a gold medal. Formal proof systems like Google’s AlphaProof are quite close to this, and even general-purpose LLMs have a fighting chance. But that's not the outcome to pay the most attention to.
@ansonwhho @ardenaberg A national AI project has become more and more of a possibility in the last year, with one as the top recommendation from a US-China congressional commission. 
First, the number of large-scale model releases is growing rapidly.
From Claude 2.0’s 100k tokens in 2023 to Llama 4 Maverick’s 10M earlier this year, there’s no doubt that context windows are getting longer. On a set of models from Artificial Analysis, we find that the longest available context windows have grown at about 30x/year.
@APotlogea @ansonwhho Skepticism around explosive AI growth often hinges on "Baumol effects"—bottlenecks from human-dependent tasks. But to their surprise, the most comprehensive integrated assessment model of AI to date suggests these constraints are weaker than expected
o3-mini-high is extremely knowledgeable, and it’s not pure memorization. It fairly reliably invokes relevant techniques and results from the mathematical literature, even when problems were designed to obscure them. 
Numerical formats tell computers how to represent numbers for calculations. Higher precision formats like FP32 use more bits, in order to store the number to a higher number of significant digits. But precision comes at a cost, as each calculation takes longer to carry out.
Our competition included around 40 mathematicians, split into teams of four or five, and with a roughly even mix of subject matter experts and exceptional undergrads on each team. We then gave them 4.5h and internet access to answer 23 challenging FrontierMath questions.
Performance has grown drastically – FLOP/s of leading AI supercomputers have doubled every 9 months. Driven by:
On FrontierMath, our benchmark of highly challenging, original math questions, o4-mini with high reasoning sets a new record in our evaluations, with an accuracy of 17% (±2%)!
On GPQA Diamond, Grok-3 is one of the top performers at 76% accuracy, beating the best competing non-reasoning models (GPT-4.5 and Claude 3.7 Sonnet) and some reasoning models like o1 and DeepSeek-R1. Grok-3 mini is slightly behind at 70 to 74%. 
For context, GPQA Diamond is a set of very difficult multiple-choice questions about biology, chemistry, and physics; human experts only score around 70%.
Historically, AI benchmarks were just designed to compare models – could a new model or technique improve on the state-of-the-art? Benchmarks focused on tasks 'just within reach' where progress was measurable.
While it’s commonly assumed that growth is mostly driven by R&D, rigorous estimates don’t agree with this intuition. For example, the BLS estimates that private R&D only accounts for around 20% of the labor productivity growth in the US since 1988.
Imagine if a central bank took AI seriously. They’d build GATE—merging economics with AI scaling laws to show how innovation, automation, and investment interact.
For example, GPT-4 cost about $40 per million tokens. 14 months later, Gemini 1.5 Flash could beat GPT-4's score on a set of Ph.D. level science questions, despite being about 300x cheaper per token. 
When you double your compute, you don't just double output by deploying more AI instances—you can also train larger, more efficient models that reduce inference cost. Doing both simultaneously creates increasing returns: output grows more than linearly. 
We can only extrapolate capabilities to the future when AI already shows some competence at them. So extrapolations often end up anchoring too much on what AI can do today, and not enough on the new tasks it will be able to do in the future.
