Super excited to share that the non-profit I spent last year co-founding (AVERI) just launched🚀

AVERI stands for AI Verification and Evaluation Research Institute, and today we published a paper in collaboration with researchers at dozens of orgs. 🧵

https://x.com/AVERIorg/status/2011852551155548335?s=20

Jeremy Kahn at Fortune just published an article on our work, which is about making third-party auditing of frontier AI effective and universal: fortune.com/2026/01/15/for…

🚨 🚨 Important Paper Alert 🚨 🚨

This is one of the most interesting analyses of AI progress in a while IMO. Check out at least the METR thread here, if not the blog post + paper.

Some additional thoughts from me below, for after you’ve read that🧵

https://twitter.com/metr_evals/status/1902384481111322929

First, I think that the long-term trend-line probably underestimates current and future progress, primarily because of test-time compute.

They discuss this a bit, but I’m just underscoring it.

The 2024-2025 extrapolation is prob. closest, though things could go faster.

Some people have asked me some variant of this question lately:

"When will we get really dangerous AI capabilities that could cause a very serious incident (billions in damage / hundreds+ of people dead)?"

Unfortunately, the answer seems to be this year, from what I can tell. Why do I say this?

Well, first of all, it's not that high of a bar. The Crowdstrike incident alone cost $10 billion+ without any fancy AI.

COVID killed tens of millions without any fancy AI.

And AI is getting very capable in these + other domains.

Lots of AI policy is based on pretraining being where the intelligence comes from, and measuring that in FLOPs.

RL on chain of thought challenges this, since it boosts intelligence materially over pretraining.

What to do if you still want to focus regs on a subset of systems? There is a short term solution and a long term one.

Short term: add pretraining and RL into a more generic “training” category (where inference FLOPs during RL are included) and also shift upward your expectation of the amount of intelligence attainable per FLOP.

RL on chain of thought (i.e. the o1/o3 series of models from OpenAI and similar ones elsewhere) is relevant to non-math, non-coding problems.

It’s nice to have an infinite source of ground truth (“is this the exact right number,” “is this proof valid”) but not essential. 🧵 1. RL on chain of thought leads to generally useful tactics like problem decomposition and backtracking that can improve peak problem solving ability and reliability in other domains.

Glad to see OpenAI sharing their latest thinking re: corporate structure, and a lot of the explanation makes sense. However, there are some red flags here that need to be urgently addressed + better explained publicly before the transition goes through.🧵

https://twitter.com/openai/status/1872628736690123213

First, just noting that I agree that AI is capital intensive in a way that was less clear at the time of OpenAI’s founding, and that a pure non-profit didn’t work given that. And given the current confusing bespoke structure, some simplification is very reasonable to consider.

I've been saying recently that completely superhuman AI math and coding by end of 2025 was plausible - 50/50 or so.

Now I'd say it's much more likely than not (o3 is already better than almost all humans). Does this mean humans will be able to add 0 value in these areas? Not necessarily - knowing the problem to solve requires insight/data from other domains, and it may be like chess/Go where there's a centaur period where humans can *occasionally* help even if weaker head-to-head.

My team + many others across OpenAI have been thinking for a while about the implications of AIs that can think for a while.

More to say later (and do read all the o1 blog posts, system card, testimonials etc. and ofc try it), but for now, 3 personal reflections. 1. Test-time compute as a new scaling paradigm has implications for governance/safety. Models can (if deployers want) have more "self-control" via thinking things through first, the capabilities attainable with a given base model are greater, + inference compute matters more.

"They're just saying that their technology might kill everyone in order to fend off regulation" seems like a very implausible claim to me, and yet many believe it.

What am I missing? Are there precedents for that tactic? How would regulating *less* help with the killing thing? I certainly can't rule out that there are some galaxy brain execs out there, but this seems pretty obviously not the reason why e.g. there was a statement spearheaded by...academics, and many people who have talked about this stuff long before they had products to regulate.

The cluster of issues around:

- Use of AI in influence operations + scams
- Watermarking/provenance for AI outputs
- Online proof of personhood

is definitely on the harder end of the spectrum as far as AI policy issues go.

Lots of collective action problems + tradeoffs. It's also among the more underestimated issues--it's *starting* to get semi mainstream but the full severity + linkages to other issues (how do we solve the other stuff if no one knows what's going on + democracy is breaking) + lack of silver bullets are not widely appreciated.

A lot of people are wondering if the current moment of interest in AI regulation is just a passing thing or the new normal.

I think it's almost certainly the new normal. The reason it's happening is widespread AI adoption, and *that* is only going to massively increase. (unless, that is, there is significant regulation to prevent it, so... 🤷‍♂️ )

Timely read in the age of #twitterdown en.wikipedia.org/wiki/I_Have_No… Naturally #twitterdown is censored lol

Some meta-level thoughts on the ChatGPT-induced alarm among teachers/professors about LM-enabled cheating (substantive thoughts to come next year):

- like many LM things, it is not coming out of nowhere. People warned about it (+ some started cheating) no later than Feb. 2019. - similarly to other LM things, the severity of the issue (and the prospective benefits for education via legitimate uses of LMs) scales with model capabilities, so was nascent before, and will be more severe in the future. Also scales with wide access + usability of interface.

Like everyone else, it seems, I have hot takes on Elon/Twitter stuff. I will try to differentiate mine by making them falsifiable.

Let’s call three scenarios for how this plays out “Exit Left,” “Mission Accomplished,” and “Golden Age.”
🧵 Exit Left (~40% chance): Mass exodus after overly aggressive relaxing of rules in a more politically conservative direction + general anti-Elon animus on the left leads …

I think some are reticent to be impressed by AI progress partly because they associate that with views they don't like--e.g. that tech co's are great or tech ppl are brilliant.

But these are not nec related. It'd be better if views on AI were less correlated with other stuff.🧵 (The premise is admittedly speculative--I am confident there's a correlation but less so re: causality. Best theory I have but will be curious for reactions. There are of course other reasons to be unimpressed such as things really being unimpressive, fear of seeming naive, etc.)

Excited to finally share a paper on what a huge chunk of OpenAI has been working on lately: building a series of code generation models and assessing their capabilities and societal implications. 🧵

arxiv.org/abs/2107.03374 First, just want to emphasize how collaborative the effort has been. Assessing the societal implications of any technology—especially pre-widespread deployment—is inherently hard and requires drawing on lots of disciplinary perspectives. See long authors list + acknowledgements!

What's going on here besides people optimizing for different things, or not bothering to do their homework? One answer is that AI policy researchers are falling victim to Collingridge's dilemma (en.wikipedia.org/wiki/Collingri…). That is, by default, people miss the sweet spot between when AI outcomes are not yet foreseeable in detail, and when a lot of the key decisions have been made. That time period is short (small number of months/years), e.g. I think language models are in the sweet spot right now.

There's been a ton of growth in publications on AI ethics/policy over the past few years. I think most observers would agree that only a small fraction of that output is "impactful," though. A thread of preliminary thoughts on what that means/why it's happening:🧵 [A note on terminology: AI ethics/policy/safety/governance etc. can be distinguished in various ways. I use "AI policy" as a catch-all for all these here because I think my points are pretty general, but I'm interested in feedback on that among other points made here!]

I mostly enjoyed "Dark Skies: Space Expansionism, Planetary Geopolitics, and the Ends of Humanity" a while back and find myself thinking about it from time to time. The big idea is that humanity expanding into space isn't as obviously good as many think. amazon.com/Dark-Skies-Exp… I'm generally pro space expansionism/the book didn't change my mind a ton but it was useful for reexamining my biases. The book is sort of an analytical companion to the show The Expanse, which also presents a less-than-utopian portrayal of space, with similar themes...

Lady Gaga as diagrams about AI systems (thread). AlphaGo.

I've been thinking a bit about the growing practice of fine-tuning generic pretrained models: first in computer vision, now NLP (highly recommend @seb_ruder's great article on this ruder.io/nlp-imagenet/)...Last time I mentioned this, people were skeptical that RL would be next. E.g. folks pointed out that RL is maybe the wrong level/domain of analysis, and maybe there is insufficient commonality across RL tasks people care about for this to make sense, etc... But in any case, I suspect more will happen in this vein beyond vision and NLP.