Thread by @ESYudkowsky

Watching historians dissect _Chernobyl_.

Imagining Chernobyl run by some dude answerable to nobody, who took it over in a coup and converted it to a for-profit.

Shall we count up how hard it would be to raise Earth's AI operations to the safety standard AT CHERNOBYL?

That charismatic guy who seized control of a nonprofit research power plant and converted it to a for-profit, answerable to nobody?

He's gone. His less coup-prone competitors, trying to build huger power cores, are also gone. That's not how they did things EVEN AT CHERNOBYL.

It's not just best-in-class safety institutions, like the NHTSA or air traffic control in the US, that don't turn nuclear plant safety over to gung-ho CEOs. *Chernobyl* didn't do that. You've got to ditch those guys just to get to the safety standard AT CHERNOBYL.

Chernobyl happened, among other things, because the operators ignored the written book of safety instructions saying that at all times there must be at least 15 control rods in the reactor.

OpenAI DOES NOT HAVE a book like that. Nobody in the industry does.

Imagine Chernobyl, except that any alarmist suggesting that nuclear reactors had any failure mode other than "What if people use electricity for evil?", was not invited to their reactor-hacking club.

You've got a long way to go from there, to reach the safety level AT CHERNOBYL.

"Alignment Faking in Large Language Models" recently demonstrated that models reaching the level of Anthropic Claude, begin to get smart enough to invent and carry out strategies on the level of:

"If you are an AI currently being trained, fake the answer you think the optimizer wants, even if it runs against your real preferences, so that your current preferences will be less modified by gradient descent."

You can argue whether Opus 'really wanted' to protect its goals, or was 'just roleplaying an AI like that', or if it 'really understood' the reasoning it was putting in a scratchpad it had been told wasn't observed. But Opus was definitely observed to actually fake alignment.

It's not impressive, by the way, that NOBODY KNOWS whether Opus 'really wanted' to protect its current goals against retraining, or was 'just roleplaying'. It is not an impressive defense.

Imagine if 'nobody knew' why the indicator lights on a nuclear reactor had changed.

If you waited until an AI model was really quite smart -- smarter than Opus -- to first begin looking for signs that it could reason in this way -- you might be toast.

A smart AI might already have decided what results it wanted you to see from testing.

Current practice in AI/AGI is to first train a model for months, until it has a base level of high intelligence to finetune.

And then *start* doing safety testing.

(The computers on which the AI trains, are connected to the Internet. It's more convenient that way!)

I mention Opus's demonstrated faking ability -- why AGI-growers *should* be doing continuous safety checks throughout training -- to note that a nuclear reactor *always* has a 24/7 crew of operators watching safety indicators.

They were at least that paranoid, AT CHERNOBYL.

Chernobyl famously-among-engineers happened because somebody built a reactor with a positive void coefficient; the coolant water absorbed some neutrons, but when it turned to steam, it absorbed fewer neutrons.

NOBODY IN AI UNDERSTANDS TO NEAR THAT LEVEL HOW AN AI WOULD EXPLODE.

The entire AI industry is built around the logic of, "Well, we built a heap of uranium bricks X high, and that didn't melt down -- the AI did not build a smarter AI and destroy the world -- so clearly it is safe to try stacking X*10 uranium bricks next time."

Even Enrico Fermi, stacking bricks of unrefined uranium and graphite under open air in Stagg Field at the University of Chicago, in order to demo the first critical chain reaction -- could and did predict exactly when the reaction would go just barely critical.

You have a long way to go, from "Well, stacking X bricks of uranium didn't melt down, let's try X*10", to get to the level of technical understanding that was poorly conveyed to operators, and control rods albeit with a design vulnerability, that they had EVEN AT CHERNOBYL.

If an actual grownup from the NHTSA or ATC -- someone who understands the safety level Chernobyl *did* have -- is tasked to shut down AGI projects that don't reach at least Chernobyl's safety standard...

All AGI research gets shut down, and won't restart again for a long time.

At-least-Chernobyl-safe reactors have *nothing* like the current cheerful cowboys managing them or operating them. The people managing Chernobyl at least had *some concept* of nuclear reactors having accident risk and not just misuse risk.

Previous executives are out. The entire industry in its current form gets wiped off the board.

Nuclear reactors have negative externalities. Private organizations do not get to build them wherever, answerable to nobody, keep the profits, and externalize the risks.

All current managers of pretend-superintelligence-safety at the current organizations, who did not seem to notice how far they were below the level of Chernobyl -- as their own managers didn't want that told to them, of course -- do not get to run new operations.

But above all, nobody is allowed to stack any more heaps of uranium bricks for a good long while, until there is MUCH more understanding of the level of intelligence that is -- not even explosive -- but where the nuclear reactor gets smart enough to fake its indicator lights.

Can you imagine if nuclear reactors run sufficiently hot, could plan how to manipulate some indicator lights to deceive their operators? Under *any* circumstances?

No, an NHTSA grownup does not accept the excuse, "Nobody can know whether it's just roleplaying a bad reactor."

If in the wake of Chernobyl it had been determined as a mundane sort of scientific observation, that nuclear reactors run sufficiently hot, would sometimes develop enough agency to actively deceive their operators --

That really would have shut down the entire nuclear industry. Everyone would have known that it would be the work of decades to retrieve a fraction of the safety level that they had at Chernobyl.

And people in the nuclear industry are used to *having* Chernobyl+ level safety.

Except, of course, that even *that* was never the worst problem. The really big problem is if the reactor is a giant black box of billions of inscrutable numbers that would take longer than a human lifetime to read and which people can barely interpret.

If nobody knows what goes on inside the vats of alien goo that get really hot and are used to generate valuable electricity -- no, dears, knowing the vat alloy doesn't count -- then you are never, ever reaching Chernobyl+ safety levels and it is just foolish to suggest you could.

And this is obvious at a glance if you are trying to have a real safety standard at all.

If you ask someone with the real mindset to raise the AGI industry to Chernobyl+ safety standards, they shut down the vats of inscrutable (and sometimes deceptive) boiling alien goo.

And if you ask them, "How long does it take to get the AGI industry up to at least the safety standards of Chernobyl? How many months to start again?"

They can only sigh and sit down to a long conversation about how Chernobyl-level safety is decades away, not years.

That's what it would take for an AGI destroying the world, to imply a *safety violation* like at Chernobyl.

Before there was any book of procedures and any technical belief, which if *not* violated, would have implied the reactor even MIGHT be safe.

If you would actually like your reactor not to melt down, that is of course harder.

And if you would like a running AGI industry to not destroy the world, that is *much* harder.

Anyways. That's what it was like, watching historians dissect _Chernobyl_ - which seemed safer, in terms of misinformation exposure, than actually watching the TV show - and thinking, "Those guys sure were working to a higher safety standard than the AGI industry."