Thread by @EpistemicHope

Ok. I've been trying to have this conversation on twitter, and its been...difficult, so far.

The nature of twitter, and the number of people involved, has caused me to neglect some of the basic moves for good conversation.

https://twitter.com/EpistemicHope/status/1342718437500485634?s=20

Including many moves that I actively teach. Embarrassing!

In particular, given the number of people responding to me I've fallen into a pattern of giving counter arguments to specific, false (in my view) claims, without checking / showing that I've understood the claims.

So (aided by @VictorLevoso's example in a private correspondence), I'm going to offer a paraphrase of my current understanding of the Crit Rat view on AI risk, in a central place where everyone can respond at once.

This is my first draft, and I'll make more drafts as needed if this version doesn't capture what's important about the key claims from your perspective.

So feel free to leave nit-picky comments.

econlib.org/archives/2011/…

Also, if you feel like this leaves out some big separate argument, you can link me to it.

One thing that is not helpful at this stage is arguing for this view, if you feel like my statement of it is solid. Let's make sure I understand it before we disagree about it.

I'm going to specifically hold of on responding to this view, until I have gotten broad confirmation that I have understood all the important parts.

ELI'S PARAPHRASE OF THE CRIT RAT STORY ABOUT AGI AND AI RISK

There are two things that you might call "AI".

The first is non-general AI, which is a program that follows some pre-set algorithm to solve a pre-set problem. This includes modern ML.

There might be some risk from powerful, but non-general, non-creative AI systems, and it seems fine to think about that some.

But it non-creative systems are not extremely dangerous: they won't be strategic creative adversaries.

The second kind of thing called AI, is what we would properly call AGI, is different.

An AGI is a program that is CREATIVE, which means that it can generate new explanatory theories as well as create and criticize ideas.

A creative entity is "universal" in the sense of "universal computation": It can comprehend any idea that can be represented by a turning machine.

Creative entities are PEOPLE.

The only thing that we know of (so far) that is creative in is individual humans.

Indeed.

The "goals" of a creative entity are ideas, in its mind, like other ideas. They're not some stable permanent thing, that is hard-coded.

Creative entities consider and criticize different possible goals, and through that process tend to change (indeed, improve) their goals over time.

That's NOT to say that the starting conditions of an agent's goals don't matter. But they aren't constraining.

A human has some innate predispositions from evolution (things like fear of heights or a propensity to violence), we can overcome those predispositions.

(An example that @iamFilos has given several times is one of learning to find sky-diving exhilarating instead of terrifying.)

Similarly, AGIs would presumably have some innate predispositions that result from the parochial details of their design process, but they would likewise be able to overcome those predispositions and change their goals.

Therefore, if a person tries "aligning" an AGI, that must mean inhibiting the ability of the AGI to criticize and change some subset of it's ideas, namely its values.

This seems like it must have the outcome of either

1) breaking the basic general creativity of the AGI mind

or

2) Creating some kind of slave-monstrosity that has the values that you chose at the time frozen in

And in fact, we already know how to live safely in a society with creative entities (people).

We raise them (lovingly), and educate them. We give them moral arguments, and have them reason for themselves about what is good to do.

We don't try to inhibit their ability to change their goals.

Soon they will be participating in the great process of figuring out which explanations are best, and helping us to generate new moral knowledge!

There IS some risk that an AI system will get "hung up" on some set of ideas for some amount of time, in the same way that humans can get hung up on a set of ideas, and this manifests as our neuroses.

If AIs are particularly powerful, this seems like it could be quite bad.

Because it might to do a lot of damage before its epistemic process got unstuck.

But, that seems like a far cry from "the default outcome is doom."

[end paraphrase]

How was that? Is that leaving out anything major?

@threadreaderapp unroll

@iamFilos
@micahtredding
@MatjazLeonardis
@dchackethal
@WorriedDenizen
@DorfGinger
@Vivify705
@krlwlzn
@reasonisfun
@DavidDeutschOxf

Feel free to call in anyone who really should be linked here.

@diviacaroline You like this kind of stuff

* hold off

https://twitter.com/EpistemicHope/status/1346774309637746689