Liveblog/commentary of that Situational Awareness Leopold Aschenbrenner piece: here goes!
situational-awareness.ai
situational-awareness.ai
I'm going to quibble with this graph -- the "preschooler"/"smart high schooler" stuff doesn't really describe what's going on at all. situational-awareness.ai/from-gpt-4-to-…
GPT-2 produced adult-sounding but illogical text.
GPT-4 answers far more questions right than any real high schooler would, but can't initiate any action spontaneously.
The basic "dramatic rapid improvement" story is correct, but "mental age" isn't a natural way to look at it.
GPT-4 answers far more questions right than any real high schooler would, but can't initiate any action spontaneously.
The basic "dramatic rapid improvement" story is correct, but "mental age" isn't a natural way to look at it.
Most of the facts in this section are due to @EpochAIResearch, which I've found to be a trustworthy organization compiling quantitative trend data on AI.
Yes, indeed, when you grow compute by about an OOM a year, you get much better LLMs very fast.
Yes, indeed, when you grow compute by about an OOM a year, you get much better LLMs very fast.
@EpochAIResearch Algorithmic efficiency: again, he's sourcing these figures from places I trust, seems fine to me.
@EpochAIResearch the data wall: could be a bottleneck, could be surmountable if AI labs learn to focus on "higher-quality" data or train in less-dumb ways.
He warns of "large error bars". So far, so reasonable.
He warns of "large error bars". So far, so reasonable.
He calls "unhobbling" all kinds of improvements that aren't training base models -- RLHF, chain-of-thought, integrations with web browsing/calculators/etc -- and notes that these can further improve the usefulness of LLMs.
Kind of a wastebasket taxon, but yeah, valid.
Kind of a wastebasket taxon, but yeah, valid.
(the very fact that "prompt engineering" is a thing, that good prompts and finetunes can perform so much better than bad ones, indicates that there's a lot of improvement that's more "mundane software engineering/product development" than "machine learning" per se.)
giving an LLM full access to a computer and treating it like a "drop-in remote worker" sounds a bit alarming from a security POV, but...ok, not in principle impossible.
This is his projection and it's pretty close to what I came up with independently over a similar timescale (unsurprisingly, given that we both relied on Epoch AI data.)
But "outperform PhDs and experts" is, again, a sloppy and nonsensical prediction.
GPT-4 can already outperform PhDs and experts at breadth of instant recall, and a pocket calculator can outperform them at arithmetic.
GPT-4 can already outperform PhDs and experts at breadth of instant recall, and a pocket calculator can outperform them at arithmetic.
So what kinds of tasks do you *really* think future LLMs are going to be superhuman at, in a meaningful and nontrivial sense?
Aschenbrenner doesn't say.
I suspect that treating all PhDs-and-experts as the same category is already a mistake.
Aschenbrenner doesn't say.
I suspect that treating all PhDs-and-experts as the same category is already a mistake.
He's also right to note that the super-fast compute scaling for AI is a 2020s thing, not a forever thing.
Compute spending growth is currently *much faster* than Moore's Law because we've "just discovered" AI is worth spending $$ on. That growth rate won't last.
Compute spending growth is currently *much faster* than Moore's Law because we've "just discovered" AI is worth spending $$ on. That growth rate won't last.
Ok, now for the wilder speculation.
Can you use a bunch of automated ML researchers to improve algorithmic efficiency by a lot?situational-awareness.ai/from-agi-to-su…
Ok, yeah, "try a simple idea from a menu of known concepts" does sound like something that can benefit from automation, especially if the available compute is vast and you can iterate a lot.
but the obvious issue here is... at that point, you're running out of benchmarks.
in this late-2020s/early-2030s world, the LLMs, by stipulation, have already maxed out on human-designed tests like the GRE, LSAT, etc.
in this late-2020s/early-2030s world, the LLMs, by stipulation, have already maxed out on human-designed tests like the GRE, LSAT, etc.
you can always continue to work on more efficient learning algorithms, that can get "good scores" using less data and less time, and that's what Aschenbrenner focuses on...
but what do you *do* with more efficiency?
but what do you *do* with more efficiency?
Aschenbrenner hopes that automated ML research will enable automated "scientific and technological progress", but I think that's probably an overly ML-centric narrative.
Developing an AI hardware engineer is probably *not* mostly about more efficient learning algorithms!
Developing an AI hardware engineer is probably *not* mostly about more efficient learning algorithms!
Again, an economic growth explosion would be *cool*, that's my ideal outcome too, but I expect that it's less something that just pops out from auto-ML and more about developing good datasets and experimental feedback loops.
also that's a pretty out-of-touch paragraph.
"arcane regulation might ensure lawyers and doctors still need to be human"??
"PERHAPS we’ll want to retain human nannies"??
no mention of regulatory barriers when he wonders why there aren't way more factories today??
"arcane regulation might ensure lawyers and doctors still need to be human"??
"PERHAPS we’ll want to retain human nannies"??
no mention of regulatory barriers when he wonders why there aren't way more factories today??
sounds like he is a bit too gung-ho about automating everything even for *me*, and has *vastly* underestimated how many things are regulated in the present-day developed world.
"doctors and lawyers must be human" is far from the dumbest or most over-cautious rule out there.
"doctors and lawyers must be human" is far from the dumbest or most over-cautious rule out there.
speedily, speedily, in our days. 😜
but seriously, this is dumb on many levels.
I'm not sure superhuman persuasion is a thing, I'm no computer security expert but the "super-smart systems can hack anything" thing sounds dumb too...
but seriously, this is dumb on many levels.
I'm not sure superhuman persuasion is a thing, I'm no computer security expert but the "super-smart systems can hack anything" thing sounds dumb too...
and most importantly,
if the thing is *literally superintelligent*, such that it can invent new weapons that work --
then nobody controls it, it controls itself.
if the thing is *literally superintelligent*, such that it can invent new weapons that work --
then nobody controls it, it controls itself.
I trust him to run his back-of-the-envelope calculations correctly...
but wtf? is anybody actually going to Do The Needful to get that much power for a single training cluster???
but wtf? is anybody actually going to Do The Needful to get that much power for a single training cluster???
Hundreds of billions of dollars spent on AI capex cumulatively isn't crazy, but the power *is*.
What, are we going to fix the grid? in four years? I wish.
(or just do off-grid solar to power a datacenter like @CJHandmer wants? maybe...)
What, are we going to fix the grid? in four years? I wish.
(or just do off-grid solar to power a datacenter like @CJHandmer wants? maybe...)
@CJHandmer this has a bunch more detail arguing why an off-grid, solar-and-battery-powered data center would be feasible -- I haven't vetted it for myself but it sounds interestingaustinvernon.site/blog/datacente…
😂
I mean, this is a valid point; it would broadly be good for the US national interest if the data centers were in America; but also this is all too familiar from the semiconductor fab/CHIPS act saga.
I know this kind of copy. I've *written* this kind of copy. It's no match for bureaucracy.
I mean, this is a valid point; it would broadly be good for the US national interest if the data centers were in America; but also this is all too familiar from the semiconductor fab/CHIPS act saga.
I know this kind of copy. I've *written* this kind of copy. It's no match for bureaucracy.
To paraphrase Gandhi:
"What do you think of computer security?"
"I think it would be a good idea."situational-awareness.ai/lock-down-the-…
This is a pretty reasonable section.
Yes, China has a large state-backed industrial espionage operation.
If it's true that Google's own assessment is that their model weights are not secure against even well-resourced cybercriminals, let alone state actors, that's...bad.
Yes, China has a large state-backed industrial espionage operation.
If it's true that Google's own assessment is that their model weights are not secure against even well-resourced cybercriminals, let alone state actors, that's...bad.
I can't evaluate his specific security recommendations, I'll throw it up to the internet to judge if these are reasonable
So, he's a superalignment guy.
He thinks ASI x-risk is worth worrying about, but tractable to prevent if we do the right R&D.
He takes a "middle path" between "superintelligence is almost-certain doom" and "superintelligence is benign-by-default."
He thinks ASI x-risk is worth worrying about, but tractable to prevent if we do the right R&D.
He takes a "middle path" between "superintelligence is almost-certain doom" and "superintelligence is benign-by-default."
This is the issue superalignment intends to deal with.
The idea is that we humans can "vet" small amounts of output to see if it's "safe", but not large amounts of more-opaque output.
The idea is that we humans can "vet" small amounts of output to see if it's "safe", but not large amounts of more-opaque output.
This is a narrow problem and I think it's misconstrued to begin with.
RLHF is only relevant to "safety" via an abuse of language -- it's used to downvote outputs containing graphic sex, copyright violations, racial slurs, etc.
RLHF is only relevant to "safety" via an abuse of language -- it's used to downvote outputs containing graphic sex, copyright violations, racial slurs, etc.
RLHF is not actually effective *even now* at preventing literally unsafe uses of LLMs;
for instance, is your autogenerated code going to cause an industrial accident?
the MTurkers doing RLHF won't know!
for instance, is your autogenerated code going to cause an industrial accident?
the MTurkers doing RLHF won't know!
And, of course, RLHF (scaled or not) simply isn't alignment in the old-school sense of "building Good Values into the seed that will grow into a nearly-all-powerful superintelligence."
Upvoting and downvoting chatbot results doesn't inculcate any sort of coherent value system.
Upvoting and downvoting chatbot results doesn't inculcate any sort of coherent value system.
but ok, let's take his narrower frame for now.
if you have a bunch of LLM agents running around mostly unmonitored, acting like "drop-in remote workers", you really don't want them to do a bunch of fraud, or wreak havoc on the computer systems you're giving them full access to.
if you have a bunch of LLM agents running around mostly unmonitored, acting like "drop-in remote workers", you really don't want them to do a bunch of fraud, or wreak havoc on the computer systems you're giving them full access to.
and yeah, maybe this is a case where an RLHF-shaped kind of post-training would prevent Bad Behaviors, but literal RHLF is too expensive to cope with the scale and opaqueness of the use case of "AI agent with full computer access"?
and that's where "superalignment" comes in?
and that's where "superalignment" comes in?
Scalable oversight does have useful applications IMO.
(arguably, the success of GANs validates that "have one model critique another model" can improve performance!)
And human-automation hybrid approaches to, e.g. check autogenerated code for bugs could be useful.
(arguably, the success of GANs validates that "have one model critique another model" can improve performance!)
And human-automation hybrid approaches to, e.g. check autogenerated code for bugs could be useful.
None of this, IMO, is relevant to the Yudkowsky-style x-risk situation, but I can see it being a useful part of the R&D effort towards making it practical to use highly autonomous AI agents all over the economy without wrecking everything.
(though, again, Aschenbrenner is an ML guy, and the solutions to the "mundane" risks of widespread AI agents are not necessarily going to be ML-based.
E.g. maybe you scaffold some old-fashioned symbolic-computation or type-checking stuff to avoid bugs in autogenerated code.)
E.g. maybe you scaffold some old-fashioned symbolic-computation or type-checking stuff to avoid bugs in autogenerated code.)
There have been "pleasant surprises" in AI development where the AI generalizes beyond what it was trained on but still gives the "correct" result in the broad situation.
I agree, it's probably a good idea to study why this happens, and when!
I agree, it's probably a good idea to study why this happens, and when!
I agree interpretability research is good.
I'm pretty shocked by his preference ordering: mechanistic interpretability is Super Real, and top-down interpretability is...a lot more prone to sloppy editorializing about We Found The Lying Neuron.
I'm pretty shocked by his preference ordering: mechanistic interpretability is Super Real, and top-down interpretability is...a lot more prone to sloppy editorializing about We Found The Lying Neuron.
Ugh. Red-teaming is good, but this field is so full of stuff that's easy to fake.
"We programmed the system to Do a Bad and it did! AI Risk is real!" and "Our extensive, totally secret, red-team eval found the system did Zero Bads! It's definitely safe!"
"We programmed the system to Do a Bad and it did! AI Risk is real!" and "Our extensive, totally secret, red-team eval found the system did Zero Bads! It's definitely safe!"
I sure hope this doesn't mean he thinks AI models should *in full generality* not be trained on biology or chemistry content...
I like being able to ask the chatbot science questions, thank you very much!!!
I like being able to ask the chatbot science questions, thank you very much!!!
I'll defer to actual computer-security people on this, but it sounds tentatively sensible.
the first thing an "actual" x-risk-level superintelligence would do is try to increase its access to compute and/or information.
seems good to make that hard and make it trigger alarms.
the first thing an "actual" x-risk-level superintelligence would do is try to increase its access to compute and/or information.
seems good to make that hard and make it trigger alarms.
This is a good point -- there have been a lot of "pleasant surprises" in the direction of "given enough data, pathological edge cases melt away and the AI basically learns to do what we mean it to do" which should update us (at least a bit) in favor of "alignment-by-default"
His case for ASI being a crucial *military* technology is basically R&D.
I think this is probably bogus. I expect that an AI needs something-like-"agency" to be an effective inventor...at which point it's not *your* military technology, it's an agent with its own goals.
I think this is probably bogus. I expect that an AI needs something-like-"agency" to be an effective inventor...at which point it's not *your* military technology, it's an agent with its own goals.
But this is a bit more reasonable. You don't need "true" superintelligence/agency for AI to be quite useful in industrial automation and R&D.
Bigger economies with stronger industrial bases plausibly will win more wars.
Bigger economies with stronger industrial bases plausibly will win more wars.
I thought i would have more skepticism about China's ability to compete on AI, but actually this is a fair point.
even if China continues to lag behind Taiwan/Korea/US/Europe at leading-edge semiconductors and SOTA AI algorithms, they can still get the world's best-performing models if they can Build Moar Datacenters and steal our model weights.
yeah...
this is a crass GIMME DEFENSE CONTRACTS ploy
If you're talking about the near-term US/China situation...the (public) US military timelines are that they expect Stuff to happen in the late 2020s.
is that enough time for the whole "radically improved general industrial productivity due to AI" thing to occur?
is that enough time for the whole "radically improved general industrial productivity due to AI" thing to occur?
given timelines for building physical infrastructure in the US, my guess is it's *not*.
so if you want to "win an AI arms race with China" in a near-term-relevant sense, the case has to be something much more like "the DOD can use LLMs for something useful right now"
so if you want to "win an AI arms race with China" in a near-term-relevant sense, the case has to be something much more like "the DOD can use LLMs for something useful right now"
now maybe it's a great idea to incorporate 2024-2027-generation AIs into more defense stuff!
but he's not making that case!
but he's not making that case!
I'm not sure I like this.
Is it just Manhattan Project envy?
does he feel out of control and imagine that the government will inject a sense of responsibility he found lacking at OpenAI?
Is it just Manhattan Project envy?
does he feel out of control and imagine that the government will inject a sense of responsibility he found lacking at OpenAI?
...but you are predicting that AI will increasingly *constitute* all of our technology and industry
that's a pretty crazy thing to just hand over, y'know???
did you have a really bad experience with Sam Altman or something?
that's a pretty crazy thing to just hand over, y'know???
did you have a really bad experience with Sam Altman or something?
ughhhh
superhuman hacking again!
please unpack the stealthy nuclear drone swarms thing??
synthetic biology is *in principle* a thing AI can enable but the Discourse around it is so bad
superhuman hacking again!
please unpack the stealthy nuclear drone swarms thing??
synthetic biology is *in principle* a thing AI can enable but the Discourse around it is so bad
he gives such short shrift to explaining why, specifically, we should believe AI is a short-term military danger
it's a stark contrast to all the careful numbers about compute growth and performance benchmarks earlier in the piece
it's a stark contrast to all the careful numbers about compute growth and performance benchmarks earlier in the piece
cards on table, I *am* a true libertarian and *do* disagree normatively
but also you don't have to be as hardcore as me to think it's crazy to nationalize a *general-purpose information technology* because "general purpose" includes military purposes
but also you don't have to be as hardcore as me to think it's crazy to nationalize a *general-purpose information technology* because "general purpose" includes military purposes
I am slightly creeped out by appropriation of WWII iconography
I'm no fan of the CCP. and I am, pretty much, a patriotic American and sympathetic to using tech to help the US military
but Szilard, Oppenheimer, etc, were *literally fighting Nazis*
it's not the same situation.
I'm no fan of the CCP. and I am, pretty much, a patriotic American and sympathetic to using tech to help the US military
but Szilard, Oppenheimer, etc, were *literally fighting Nazis*
it's not the same situation.
this section is worse than I expected.
I thought we were going to have some garden variety shilling for defense contracts. "American Dynamism" and all that.
plus some "let's have more security" stuff the AI labs would have to do anyway to meet military specifications.
I thought we were going to have some garden variety shilling for defense contracts. "American Dynamism" and all that.
plus some "let's have more security" stuff the AI labs would have to do anyway to meet military specifications.
nope! he wants to nationalize and potentially cripple a nascent industry to prepare for a war that hasn't even started.
look
everything really awesome (super-productive manufacturing! automated R&D!) and concerning (new WMDs!) about AI
flows through a very serious "schlep" of integrating AI with the physical world.
everything really awesome (super-productive manufacturing! automated R&D!) and concerning (new WMDs!) about AI
flows through a very serious "schlep" of integrating AI with the physical world.
not that this is impossible! but it doesn't seem to be part of Aschenbrenner's world model.
the adoption curves of manufacturing companies. the *many* organizations that *do not yet exist* that would need to
the adoption curves of manufacturing companies. the *many* organizations that *do not yet exist* that would need to
when he's talking about AI models and the surrounding software tooling and data centers and such, he's concrete and detailed
when he's talking about the other stuff...he's not
it feels like he's assuming away complexity in everything that's not his field
when he's talking about the other stuff...he's not
it feels like he's assuming away complexity in everything that's not his field
there's a bit of a good point here.
for AI to be a big deal economically, it will have to "grow up" and interface with the security/safety/reliability practices of sectors like healthcare, manufacturing, transportation, etc. Lots of schleps ahead, human as well as technical.
for AI to be a big deal economically, it will have to "grow up" and interface with the security/safety/reliability practices of sectors like healthcare, manufacturing, transportation, etc. Lots of schleps ahead, human as well as technical.
but the role model here is *software*, which had to do all those things too; each industry had to get to the point where it (and its regulators) were ok with certain things being digitized and automated.
We didn't need to nationalize the software industry, even in our real-world, mixed-economy rich democracies! Even though software has military applications!
I am a long way from a defense expert, but Aschenbrenner does not "smell" like a guy who has extensive knowledge and good sense in the field.
He sounds like he's maybe interfaced with the US security world as far as encountering ITAR compliance at a tech company, and has read history books and maybe met some fancy DC people
but not like he has experience/intuition about how contracting actually works?
but not like he has experience/intuition about how contracting actually works?
it's just a guess on my part, but my expectation is that if you want to make AI For Defense a thing, there are probably better people than him to lead those initiatives
Overall takes:
1.) Aschenbrenner's views are colored by where he comes from.
He comes out of Emergent Ventures, his new VC firm is funded by the Collisons, that's the rough intellectual cluster he's from.
1.) Aschenbrenner's views are colored by where he comes from.
He comes out of Emergent Ventures, his new VC firm is funded by the Collisons, that's the rough intellectual cluster he's from.
That means: his home turf is economics and AI startups.
*Not* government or non-SV-software business (or academic science, though he's certainly technically literate.)
*Not* government or non-SV-software business (or academic science, though he's certainly technically literate.)
2. Aschenbrenner is a Silicon Valley Startup Guy who *fears his home culture can't handle AI responsibly*.
Maybe he has firsthand experience that gave him cause for such worry!
Maybe he has firsthand experience that gave him cause for such worry!
I did see elsewhere claims that his concerns about Chinese espionage were dismissed by OpenAI as "racist."
I can see how that would radicalize someone!
(It is public record that the CCP engages in espionage, like many other nations; this is not about ethnicity!)
I can see how that would radicalize someone!
(It is public record that the CCP engages in espionage, like many other nations; this is not about ethnicity!)
But I see a lot of... idealization of USG leadership and hope that someone wiser will take the reins, more based on WWII history books than any details about the present day.
I don't pretend to know how Washington DC works, but I don't believe Aschenbrenner does either.
I don't pretend to know how Washington DC works, but I don't believe Aschenbrenner does either.
3.) My guess is he's sincerely concerned about AI risk.
I thought I was going to be reading a pitch for defense funding. This was not that.
It was an almost embarrassingly heartfelt expression of worry. (And very little detail on the "DOD should buy more AI and here's why" pitch.)
I thought I was going to be reading a pitch for defense funding. This was not that.
It was an almost embarrassingly heartfelt expression of worry. (And very little detail on the "DOD should buy more AI and here's why" pitch.)
It sounds like he's not seriously engaging much with "what if the AI has goals of its own that endanger us" (classic Yudkowskian x-risk) but is very seriously concerned about a superintelligent CCP.
Also I bet he has a sense that Sam Altman, in particular, is about to wield power on a nation-state-relevant scale and he *does not want that*.
That could be a sort of "playing small" attitude ("people like us aren't supposed to be *that* powerful, dammit!")
or more of an individual thing ("this man in particular is not to be trusted with vast power!")
or more of an individual thing ("this man in particular is not to be trusted with vast power!")
Either way I'm very curious what the real backstory is.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
