Thread by @ESYudkowsky

@wolftivy @willdjarvis The scenario you generate is, in two critical ways, so unlike the real world that the answer to your scenario wouldn’t generalize to the real world. One, I expect that any "true" AGI (as it would appear in the laboratory) scales to superintelligence by running deeper and wider.

@wolftivy @willdjarvis Second, AI is not currently on a trend to be controllable or alignable at all, at the superintelligent level. If alignable at all, it will be barely, narrowly alignable, on some task chosen and constrained to a bare minimum of ambition for alignability.

@wolftivy @willdjarvis So in the real world, when you "have" AGI first, what you mostly have is a non-survivable planetary emergency where you cannot scale it and then have it do anything nice, and as soon as somebody else steals or reproduces the code and scales it, everybody dies.

@wolftivy @willdjarvis If instead you can align it all, you should do the bare minimum alignable thing, requiring as little superintelligence as possible, that averts the scenario where the world ends in 1 month or 6 months when somebody steals the code or duplicates the research feat.

@wolftivy @willdjarvis However, it's not easy to not have somebody else destroy the world. If you look at the current field, it's full of people who'd think of AGI as a pot o' gold to turn up the dial on. The equivalent of nuclear nonproliferation is a lost cause; GPU farms are everywhere.

@wolftivy @willdjarvis So you've got to do some relatively *powerful* thing with your AGI, that humans could not do or nearly do - if there was a solution with human-level intelligence, I'd be doing it right now, not waiting on AGI to do something I could have as easily tried myself.

@wolftivy @willdjarvis Everything like this one could do with a barely alignable AGI, that would actually in real life prevent the world from ending 6 months later, is outside the Overton Window; and everything inside the Overton Window is an ineffectual fantasy of political impossibilities.

@wolftivy @willdjarvis But as an example that isn't the correct answer, "Build self-replicating nanomachines and destroy all GPUs." It's not the correct answer because "nanomachines operating in open world" is too large an alignment problem. But it would prevent the world from ending in 6 months.