Thread by @ESYudkowsky

@paulg There once was a startup that had no business plan. "Why worry?" they said. "You can't predict the future. Who, in 2018, expected Covid-19? Our future business difficulties are impossible to correctly imagine. Why pretend we can?"

"A business plan is not *the* future," 1/

@paulg ...replies Paul Graham. "A business plan checks model consistency: is there a plausible world where success is possible? And a business plan writes out your assumptions; which matters, not because every assumption will be true, but so you notice if one is turning out false." 2/

@paulg (Of course, in real life, you've already decided not to fund the startup at this point. They're obviously quite doomed. It's not like you've got no choice but to bet the fortunes of your entire extended family on repairing this particular startup. But still, let's continue.) 3/

@paulg "Well, that may be *a* use of a business plan," replies the startup founder. "And I suppose that would be nice to have. But why are you so sure we're doomed without it? Truly the future is very impossible to predict; how, then, can you possibly know we'll get a bad outcome?" 4/

@paulg "That's like asking how I can know you won't win the lottery, since the future is uncertain and either you win or you don't," replies a now suspiciously Eliezer-like model of Paul Graham, using my favorite metaphor since I don't know Paul's. "Even if..." 5/

@paulg "...you admit to being uncertain, there's a question of how to slice the possibility space into outcomes you claim to be ignorant about. There's millions of lottery combinations to be unsure of, and only one winner. Some forms of ignorance look a lot like gloomy knowledge." 6/

@paulg "But our technology is different!" says the founder. And here we must depart from the metaphor, because their technology *is genuinely* different; so different that you could ask why not analogize "publishing a scientific discovery with no written plan", or "raising a baby". 7/

@paulg And so we ask the fundamental question of rationality: "What do you think you know, and how do you think you know it?" There's many specific things we can't reasonably know about the future. Are there some big, relevant things we *can* guess? And argue in 240 characters? 8/

@paulg (And the answer is: mostly no to the 240 characters part, especially if you want any countercounterarguments to common counterarguments. But hey, half the reason any good writer stays on Twitter is the ludicrous challenge of fitting arguments into 240 characters.) 9/

@paulg Here's one idea that a lot of thinkers in this field, especially computer scientists, ended up agreeing on, even if they disagreed about many other points:

If you think of a mind as a cognitive engine that outputs actions steering the future somewhere, there's many... 10/

@paulg ...possible directions it could steer the future *to*. Or to add on some extra (but defensible) ideas: there's a combinatorially vast space of possible agents with widely different utility functions, including extremely intelligent such agents. 11/

@paulg (If you object to this, I have to know which objection, in order to try to answer in 240 characters or less. But see arbital.com/p/orthogonalit….) 12/

@paulg This idea already suggests a kind of work that might be necessary to get a good outcome for AGI. In the metaphor, it's like observing that even a startup with genuinely different technology still has expenses, and still needs revenue from somewhere. If you're "unsure"... 13/

@paulg ...about what kind of AI you get, maybe you should be unsure about whether it's a staples maximizer, or a paperclip maximizer, or a tiny-molecular-smileyface maximizer, etc. Just like being unsure if the lottery combination will be 1-2-3-4-5-6, or 1-2-3-4-5-7, etc. 14/

@paulg But suppose that the work to narrow down that space isn't done; and the outcome is an AGI that wants a weird thing. Is that necessarily bad? This is another place where we can ask "Is there anything we *can* know or strongly guess?", even though the future is uncertain. 15/

@paulg In this case, that base idea that many ended up agreeing on is "instrumental convergence". If you think of a cognitive engine that takes in sensory input, builds a model of the world, and outputs actions to steer the world, then some waypoints/strategies may look similar... 16/

@paulg ...across many different places the engine could be steering the future *to*, and many different options it might access for steering it. Matter and energy are useful for making staples *or* paperclips; our prediction that the cognitive engine instrumentally... 17/

@paulg ...pursues strategies for gathering matter and energy isn't very much dependent on what the lottery turns up, if it's a lottery with equal possibilities across staples and paperclips. arbital.com/p/instrumental… 18/

@paulg Does this argument suffice to establish doom? Of course not. Many human beings who want weird things do useful-to-others work in the global economy and trade for the weird things they want. Does *that* argument establish safety? Only if you stop thinking when you get... 19/

@paulg ...an answer that makes you feel safe. Human beings have fellow-feeling by both instinct and culture; we tip in restaurants we'll never visit again. Human beings don't have power disparities great enough to offer "rewrite the other person's atoms" as an option vs trade. 20/

@paulg I could try extending these 240-character argument-units further, to cover "Would a superintelligent paperclip maximizer with a vast power disparity vs. humanity decide to trade with us?" being guessable. And more controversially, start to consider what we can know about... 21/

@paulg ...how fast cognitive intelligence might *scale* with more computing power and optimization. People who come up with comfortable-sounding guesses and then halt thought, for example, often say "Ah but maybe intelligence increases logarithmically with compute." We can know... 22/

@paulg ...this is false because hominid brain sizes were increasing while we evolved more general intelligence, which by the logic of evo-bio implies that the *marginal* returns to fitness, on brain size, were *increasing* over this evolutionary epoch. But maybe that wasn't... 23/

@paulg ...the point that you cared about in the first place, and in Twitter I can't try to answer all arguments in advance. But hopefully I've given you reason to hope that there are more argument-units like this. And the place where those further arguments lead up to... 24/

@paulg ...is this: Despite the future being impossible to predict, there's a sense in which our total inability to tell what GPT-3 is thinking, is a very bad sign. Like a startup having no business plan; in a world where, like a startup getting revenues above irreducible... 25/

@paulg ...expenses, there is work that must be done somehow to survive. There is no published plan for living through this. I doubt any secret such plans would stand up to criticism. And we can guess in advance that's a bad sign, before it all blows up. 25/END.