Thread by @HZoete

“Genesis” by Henry Kissinger, @ericschmidt & Greg Mundie faces up to the prospect of superintelligence & asks the hard questions about what its arrival means. The security chapter, which @nfergus says is “the book’s most powerful section”, is eye opening. Some quotes & thoughts:

“If, following from the instinct for self preservation, each human society wishes to maximise its unilateral position, then the conditions would be set for a psychological contest between rival military forces & intel agencies, the like of which humanity has never faced before.

Today, in the years, months, weeks, & days leading up to the arrival of the first superintelligence, a security dilemma of existential nature awaits.” p112

“No real contender for superintelligence could be blamed for acting from a position of maximum paranoia & suspicion.” p115

“Defaulting to worst case scenarios, the strategic calculus of any actor at the frontier would be to prioritise speed and secrecy over safety.” p115

“In the event that the identity of a winner does crystallize, mere competition could devolve into conflicts driven by desperation and fear. In such an instance, certitude could be even more destabilizing than precaution.

Put together, perfect deterrence with unlimited speed and maximum accuracy translates to total victory … some states may seem the advent of AI threatening enough to demand a nuclear response.” p116

“Though we may not yet have entered such a moment, preparations should be made in advance to manage the existential competition of the AI age & its attendant risks. A determined or desperate actor with secondary but sizeable strength will be vigilant in the watch for superior AIs

If that secondary actor perceived, even mistakenly, that the dominant other is within grasp of a totalizing capability, it may launch preemptive electronic or physical strikes,

setting off a cycle of unimaginable escalation & retaliation & spiralling possibility of mutual destruction…

Global leaders have a window of opportunity to engage in discussions on the basis of humanity’s collective survival.” p117

@nfergus ask in his foreword: “if we are already witnessing ‘a competition to reach a single, perfect, unquestionably dominant intelligence’, then what are the likely outcomes? The authors envision six scenarios, by my count, none of them enticing.”

Kissinger was so worried he took action in the twilight days and months of his life as @nfergus says:

The exponential improvement in AI continues.

Superintelligence is coming.

It will have profound geopolitical and national security consequences.

World leaders must acknowledge and face up to these hard questions.

They can’t be ignored for long.