Thread by @Delafina777

I was thinking the other night, while watching Person of Interest, a show I love dearly, about our apparent inability to imagine truly benevolent, all-powerful AI, just as we seem unable to imagine true utopias.

Like, POI almost gets there. It has the Machine (figured, interestingly, as feminine), but spends a lot of time positioning the Machine as morally ambiguous. The Machine's inherent goodness and compassion only are codified when she's largely defeated by the evil AI Samaritan.

Samaritan is positioned as evil largely because it wants to eliminate the show's protagonists, as it eliminates anyone deemed disruptive to society.

For the first 3 1/2 seasons, of course, the protagonists have been eliminating people THEIR AI deems harmful.

Samaritan's supporters believe in the promise of putting a superintelligent AI in charge of society: that it will rationally, unemotionally, logically do what is best for the largest number of people.

But ultimately, the show's vision for what that means is largely limited to the AI eliminating terrorists and murderers, and not much else.

It's always weird to me that we have this fiction that imagines an AI-run society in which the AI supposedly functions for the good of humanity, but we never seem to dare to imagine what that would actually look like if it, y'know, WORKED.

AIs never seem to immediately ensure everyone has food, shelter, and healthcare, for instance, which would presumably be Step 1 for logically reorganizing society for the benefit of humanity.

And I think this is because if we actually imagined something all-power, benevolent, and in charge, it would call into question the value of freedom on its own merits.

We've never really had to ask that question, since we've never had an omnipowerful and, for lack of a better term, omnibenevolent force in charge of our society.

Like, the value of freedom has always been borne out because *whoever has power abuses it.* Freedom is good because it allows us to protect ourselves against the abuses of others.

And that's always true in terms of human rule.

But like, what if there were a superintelligent AI that was *actually* good?

Would freedom actually have value *in itself* if there was nothing to protect oneself against?

Interestingly, the one fictional portrayal I can conjure up of a *genuinely* benevolent AI is in Neal Shusterman's Scythe series.

amazon.com/Scythe-Arc-Nea…

This series of books imagines a society run by a genuinely benevolent, compassionate AI. However, that AI has left the questions of reproduction and death in the hands of humanity.

Everyone remains young and healthy basically forever, *except* a certain number of people must die every year, and who dies and when is left to a human order, known as Scythes.

(Interestingly, the AI also leaves openings for ways for humanity (especially teenagers) to "rebel" in forms that are ultimately harmless. Certain types of rebellion garner one a temporary "Unsavory" status, which means the AI ignores you for a period.)

Of course the human Scythe order almost immediately becomes corrupt, leaving the AI with a moral dilemma--its mandate is the good of humanity, which involves being hands-off with the Scythes as an area of human autonomy...

...but now a threat to humanity is emerging from that area of freedom.

Its first approach to dealing with that is to place it in the hands of other humans to fix, which means withdrawing its guidance (and even contact) from them.

There's a Jewish concept called tzimtzum that seems relevant here: the withdrawal of the divine to make space for humanity's existence, free will, etc.

The second book in the series contains this moment, in which one of the humans in whose hands Thunderhead, the AI, has placed the responsibility for fixing the problem, has a crisis of faith because that freedom comes with an absence of contact with the AI.

I think this passage is gorgeous, honestly, and more spiritual than a thousand trite sermons.

But again, it's fascinating how, like POI, this narrative seems to only really *show* the benevolence and the compassion of the AI when it's in a position of helplessness, when it's *disempowered.*

And again, I think that's because it's actually a kind of scary question to wonder if freedom itself has any inherent value when the alternative is an *actual* utopia (and not one that's built on the hidden suffering of an underclass).

The only real artistic grappling I can think of with that question is Arthur C. Clarke's "Childhood's End," in which he posits that in a true utopia, humans would stop making art.

But *even there,* there is a looming "threat" in the sense that the utopia is a temporary one that exists while the last remaining humans die out and their enlightened, super-human offspring mature to the point where they can leave Earth (and their parents) behind.

It's not *really* a utopian narrative--it's an apocalyptic one.

(I'm reminded here of the difference between rabbinic characterizations of the period of Israelite history portrayed in Numbers, versus the portrayal in the actual text.

According to the rabbis, the years of wandering in the desert are one of incredible, ecstatic closeness to the divine, a honeymoon period for God and Israel.

The text of Numbers itself is the story of people stuck in a holding pattern until an entire generation dies off.

For me, the actual text is very much a horror story. How would it feel to know that there's a beautiful future out there for your children, but *your continued existence* is the roadblock to them getting there?

You can reconcile those two views of the story, I think, by viewing that period as a sort of hospice--on one hand, it is a period of waiting for people to die off, but it's sweetened by the divine presence being immediately present in a way it never will be again.

"Childhood's End" makes an interesting midrash on that concept.)

So yeah, ultimately, I don't see even "Childhood's End" as being able to imagine a genuine utopia initiated and maintained by a functionally omnipotent, benevolent force.

There's a whole other thread here about how the freedom-is-the-highest-good-for-its-own-sake attitude interfaces with American resistance to things like socialized medicine, but that's a different thread.

Here I'm just interested in how it plays out in fiction.

And yes, everyone, I've read most of Iain Banks's Culture series, but the Culture still seems to be Utopia-covering-a-lot-of-dystopian-stuff to me.

And also, yes, I recognize that storytelling usually involves conflict and it would be hard to tell interesting stories in an actual utopia, which doesn't mean it's impossible to *portray* a genuine utopia.