I was thinking the other night, while watching Person of Interest, a show I love dearly, about our apparent inability to imagine truly benevolent, all-powerful AI, just as we seem unable to imagine true utopias.
Like, POI almost gets there. It has the Machine (figured, interestingly, as feminine), but spends a lot of time positioning the Machine as morally ambiguous. The Machine's inherent goodness and compassion only are codified when she's largely defeated by the evil AI Samaritan.
Samaritan is positioned as evil largely because it wants to eliminate the show's protagonists, as it eliminates anyone deemed disruptive to society.

For the first 3 1/2 seasons, of course, the protagonists have been eliminating people THEIR AI deems harmful.
Samaritan's supporters believe in the promise of putting a superintelligent AI in charge of society: that it will rationally, unemotionally, logically do what is best for the largest number of people.
But ultimately, the show's vision for what that means is largely limited to the AI eliminating terrorists and murderers, and not much else.
It's always weird to me that we have this fiction that imagines an AI-run society in which the AI supposedly functions for the good of humanity, but we never seem to dare to imagine what that would actually look like if it, y'know, WORKED.
AIs never seem to immediately ensure everyone has food, shelter, and healthcare, for instance, which would presumably be Step 1 for logically reorganizing society for the benefit of humanity.
And I think this is because if we actually imagined something all-power, benevolent, and in charge, it would call into question the value of freedom on its own merits.
We've never really had to ask that question, since we've never had an omnipowerful and, for lack of a better term, omnibenevolent force in charge of our society.
Like, the value of freedom has always been borne out because *whoever has power abuses it.* Freedom is good because it allows us to protect ourselves against the abuses of others.
And that's always true in terms of human rule.

But like, what if there were a superintelligent AI that was *actually* good?

Would freedom actually have value *in itself* if there was nothing to protect oneself against?
Interestingly, the one fictional portrayal I can conjure up of a *genuinely* benevolent AI is in Neal Shusterman's Scythe series.

amazon.com/Scythe-Arc-Nea…
This series of books imagines a society run by a genuinely benevolent, compassionate AI. However, that AI has left the questions of reproduction and death in the hands of humanity.
Everyone remains young and healthy basically forever, *except* a certain number of people must die every year, and who dies and when is left to a human order, known as Scythes.
(Interestingly, the AI also leaves openings for ways for humanity (especially teenagers) to "rebel" in forms that are ultimately harmless. Certain types of rebellion garner one a temporary "Unsavory" status, which means the AI ignores you for a period.)
Of course the human Scythe order almost immediately becomes corrupt, leaving the AI with a moral dilemma--its mandate is the good of humanity, which involves being hands-off with the Scythes as an area of human autonomy...
...but now a threat to humanity is emerging from that area of freedom.

Its first approach to dealing with that is to place it in the hands of other humans to fix, which means withdrawing its guidance (and even contact) from them.
There's a Jewish concept called tzimtzum that seems relevant here: the withdrawal of the divine to make space for humanity's existence, free will, etc.
The second book in the series contains this moment, in which one of the humans in whose hands Thunderhead, the AI, has placed the responsibility for fixing the problem, has a crisis of faith because that freedom comes with an absence of contact with the AI.
I think this passage is gorgeous, honestly, and more spiritual than a thousand trite sermons.
But again, it's fascinating how, like POI, this narrative seems to only really *show* the benevolence and the compassion of the AI when it's in a position of helplessness, when it's *disempowered.*
And again, I think that's because it's actually a kind of scary question to wonder if freedom itself has any inherent value when the alternative is an *actual* utopia (and not one that's built on the hidden suffering of an underclass).
The only real artistic grappling I can think of with that question is Arthur C. Clarke's "Childhood's End," in which he posits that in a true utopia, humans would stop making art.
But *even there,* there is a looming "threat" in the sense that the utopia is a temporary one that exists while the last remaining humans die out and their enlightened, super-human offspring mature to the point where they can leave Earth (and their parents) behind.
It's not *really* a utopian narrative--it's an apocalyptic one.
(I'm reminded here of the difference between rabbinic characterizations of the period of Israelite history portrayed in Numbers, versus the portrayal in the actual text.
According to the rabbis, the years of wandering in the desert are one of incredible, ecstatic closeness to the divine, a honeymoon period for God and Israel.

The text of Numbers itself is the story of people stuck in a holding pattern until an entire generation dies off.
For me, the actual text is very much a horror story. How would it feel to know that there's a beautiful future out there for your children, but *your continued existence* is the roadblock to them getting there?
You can reconcile those two views of the story, I think, by viewing that period as a sort of hospice--on one hand, it is a period of waiting for people to die off, but it's sweetened by the divine presence being immediately present in a way it never will be again.
"Childhood's End" makes an interesting midrash on that concept.)
So yeah, ultimately, I don't see even "Childhood's End" as being able to imagine a genuine utopia initiated and maintained by a functionally omnipotent, benevolent force.
There's a whole other thread here about how the freedom-is-the-highest-good-for-its-own-sake attitude interfaces with American resistance to things like socialized medicine, but that's a different thread.

Here I'm just interested in how it plays out in fiction.
And yes, everyone, I've read most of Iain Banks's Culture series, but the Culture still seems to be Utopia-covering-a-lot-of-dystopian-stuff to me.
And also, yes, I recognize that storytelling usually involves conflict and it would be hard to tell interesting stories in an actual utopia, which doesn't mean it's impossible to *portray* a genuine utopia.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Jessica Price

Jessica Price Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @Delafina777

25 Feb
seeing men say "I have only daughters so my name dies with me" is like

I dunno, if it matters to you, maybe change cultural norms around women taking their husbands' names when they marry
like I dunno, I kind of think couples should come up with a new last name when they get married
or we could have the norm be hyphenating last names on marriage, and when kids reach 18 they formally and legally pick which last name (which parents', or a hyphenated name) to use as their legal name
Read 5 tweets
25 Feb
people: heya so the bible has some really nasty stuff in it

Christians: it is a product of its time, you cannot judge it by modern standards

people: okay cool then let's stop treating the outdated parts, like the anti-gay stuff, as canon

Christians: IT IS TIMELESS TRUTH
Jews: it's 2021, we know Jesus exists, maybe stop evangelizing us

Christians: THE NEW TESTAMENT IS TIMELESS TRUTH, FOREVER APPLICABLE

Jews: "in Jesus there is neither Jew nor Gentile" doesn't sound so great after Auschwitz

Christians: YOU CANNOT JUDGE IT BY MODERN STANDARDS
like, pick one

either this text is a product of its place and time and authors and context, and that means that maybe we admit some parts of it belong in the past

or it's perfect, eternal and timeless, in which case it's answerable to our ethics as well as those of its origin
Read 8 tweets
24 Feb
So I developed tendonitis in my ankle/calf/knee from either a possible stress fracture in my foot or maybe from when I broke my toe this summer or possibly from the weird extra bones in my ankle or possibly from nothing at all.

(Modern medicine is VERY precise.)
And so my trainer is trying to help me deal with it, so we've been working on ankle stability and other stuff a lot.

And today after my workout he used one of those poundy massagers on my calf and HOLY SHIT, dear reader, I almost blacked out.
(like, not gonna lie, I had no idea those things actually had a real use--I hear "massager" and assume "vibrator")
Read 5 tweets
24 Feb
*perky corporate voice*

You don't need health care when you have SELF care!!!
Hello, [insert company name] rockstars! It's been an incredible year as we made record profits and refined our processes and team to be Lean (TM) and mean. We want to thank you for all your hard work and tell you about some innovations to your benefit plan you're going to love!
We want to give you the freedom to choose your own ways to stay healthy rather than being confined by the *air quotes* "preventative care" options offered by our old plan, so we've upgraded from our former Premium Plan to a Flex Option with a creative solution called an "HSA."
Read 10 tweets
23 Feb
I also want to say that the coolest thing about working on this has been feeling pulled into the excitement that @the_strix, @ajitgeorgeSB, @GoblinPrincete and some of the other authors on it have for its launch.

Like (selfishly, I admit) it's been really healing for me.
Like, it's so easy when you've worked in TTRPGs fulltime for somewhere like Paizo, where we were putting out SO MANY products every month, to just have all the joy sucked out of having contributed.
And especially with Paizo, because there was *so much trauma* for so many people on the team. Like, even the books I poured my heart and soul into, the launches felt muted because the process of getting them out had often been ugly.
Read 10 tweets
23 Feb
So, a couple elephants in the room to address about the new Ravenloft book, as much as I can without breaking NDA.
First off, as far as I can tell, Mike Mearls was in no way involved with this book. I don't know what he's working on at WOTC now, but it doesn't seem to be print D&D.
Like, if you're boycotting all things WOTC because of him, go your way in peace.

If you're specifically boycotting D&D books, I dunno for sure, but I think you're boycotting the wrong department at this point.
Read 14 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!