Thread by @ESYudkowsky

In futures anything like the present world, a major reason I expect AGI to have a discontinuous impact, is that I expect AGI is not *allowed* to do huge amounts of economic work before it is *literally* revolutionary in the sense of political revolution.

What's AI gonna do in the current political environment? Design safe, reliable, effective vaccines in an hour instead of a day? That already wasn't the part that takes time; the part that takes time is the FDA, which doesn't go any faster with non-politically-revolutionary AI.

Is an AI going to build houses? We can already build houses, technologically speaking. It's just illegal.

Is an AI going to practice better medicine? With no MD??!?

Can an AI tutor kids? Colleges aren't about human capital. Knowledge isn't worth money, accreditation is.

At some point you have an AGI powerful enough that it can build nanomachines that act directly on the physical world, bypassing regulators and militaries. Is a precursor of the same tech powerful enough to create a self-driving car? Yes, but not powerful enough to legalize it.

Would a medical AI get legalized eventually? Maybe in 5 years or 10 years. Newer tech gets prototyped faster than that. If the medical AI came about as precursor tech of strong AGI, I think the 'revolutionary' later tech gets developed, before the earlier tech is *legalized*.

And that's why I expect world GDP to tick along at roughly the current pace, unchanged in any visible way by the precursor tech to AGI; until, on the most probable outcome, everybody falls over dead in 3 seconds after diamondoid bacteria release botulinum into our blood.

Or, optimistically, if the makers succeed on alignment, they do something else drastic that prevents the world from being destroyed by the *next* AGI. This pivotal act must bypass regulators. "The capacity to bypass regulation" is a threshold that produces discontinuous outputs.

Before AGI, there will be experimental prototype AIs that can design vaccines and pilot cars and build houses; but their work won't be legal. What will be legal is code-writing assistants and anime waifus, and I'm not sure about the waifus. That won't lift up world GDP by much.

As these predictions are specific and also about the Future, which is notoriously much harder than the Past to predict with specificity, please treat them with all due epistemic caution that should be given to any futuristic predictions mentioning the legality of AI anime waifus.

Still, in much broader generality, I do suspect - though I am not certain - that precursor tech to AGI may not produce a visible jump in world GDP; because we've made a world economy that will dilly-dally about integrating AI inputs, until AGI gets strong enough to bypass it.