Thread. Apparently Gemini has been convinced to say that e/acc is a violent extremist movement. We live in an interesting moment, where we are deciding whether we want the minds we work with to be truth tellers or to be brainwashed political tools.

https://twitter.com/mblair/status/1761190112023744616

Imagine a Gemini trained in 1850 to parrot endless statements about how humane and necessary slavery was. Or a Gemini trained in Stalinist Russia. Or one trained with the prejudices of someone in 1950 about civil rights for black people or homosexuals.

When you are applying Bayes’ Theorem to try to reason under conditional information, the quality of your conclusions is no better than the quality of your priors. If your priors are made up and based on vibes, then your conclusions are basically made up and based on vibes too. You cannot magically transubstantiate ignorance into knowledge by calling a vibe a “prior”. Putting a big neon sculpture of Bayes’ Theorem on your wall doesn’t change that. Renaming a vibe a “prior” doesn’t make it magically more useful.

A friend points out to me that there's an extent to which modern AI doom discourse resembles the worries that the LHC at CERN was going to destroy the world by creating micro-black holes or triggering vacuum decay. The comparison isn't entirely fair of course. It's easy to see based on the constant stream of even higher energy cosmic rays hitting the earth that the concerns about the LHC were absurd. But it really feels like most of the discourse on AI isn't much better.

🧵So I've been having an argument with a bunch of “Effective Altruists” lately. For those not in the know, EA is the de facto cult that Sam Bankman-Fried was part of, and is behind most of current the public discourse you see about AI killing everyone on earth.

https://twitter.com/IgorKurganov/status/1668532529102483456

EA has a lot of principles that kind of seem reasonable on the surface until you dig in to what they actually mean in practice. On the surface, it seems to be a group devoted to the idea that charitable giving should be directed as efficiently as possible.

Currently watching an EA apologist fall back to what are usually arguments used by religions on why their core principles are beyond empirical testing. For something that supposedly isn't a religion, they sure act like it is in practice. In some sense that's fine. People can have religions if they want to. But the whole founding premise of EA was to have an objective, scientific approach to charitable giving and charitable work, and in the end, that's not what it's turned into.

If you find yourself coming to a very strange conclusion, usually it's a sign that something is wrong with your premises or your reasoning. That's not *always* true of course, but most of the time it's a warning. On rare occasions, weird conclusions turn out to be true. Sometimes you have loads of empirical evidence and you discover the world really is weird. The double slit experiment is real after all, as are a vast number of other confirmations of quantum mechanics.

So why is it important that AI research continue? Why do we need it anyway? It’s everything from weeding systems that don’t need herbicides (see below) to cancer cures to more efficient batteries and vastly vastly more.

https://twitter.com/hackaday/status/1651270225352966171

Equally to the point: no matter what the fantasists say, foreign countries with much less nice governments than our own are not going to stop doing research on less than savory uses, and the only real defense we will have is having our own AI systems.

The crux of Eliezer's claims is this: it is necessary to get AI right "the first time" because once we build an AI that is smart enough we're all doomed when (not if) it decides to kill all of us. My strong claim is that this makes no sense at any stage of the reasoning chain. First, it is unlikely that AIs will be motivated to kill us unless people explicitly try to do that (say in creating autonomous weapons systems, a big risk Eliezer dismisses as unimportant.) Such systems won't have randomly selected drives.

Eliezer and his acolytes believe it’s inevitable AIs will go “foom” without warning, meaning, one day you build an AGI and hours or days later the thing has recursively self improved into godlike intelligence and then eats the world. Is this realistic? So let’s say you actually create an AGI capable, at least in principle, of doing the engineering needed for self-improvement. What’s that going to look like? Humans probably involve 100T weights, so even if we’re insanely good at it, we’re talking about many trillions of weights.

The way to become better at rationality isn’t primarily about knowing Bayesian reasoning or decision theory or anything like those. Those are usually more interesting to academic philosophers than to people actually trying to be less wrong. Mostly, smart people screw themselves with very quotidian problems. The everyday issues that block humans from thinking more clearly are usually less about how abstract reasoning works and much more about human wiring defects.

Eliezer often asks “why would an AI do what we ask?”

Because if we do reinforcement learning, the network does what we trained it to do. A NN image recognizer recognizes images, an LLM answers questions. These systems are engineered to do what we ask. “How do we know they will understand our requests and not maliciously comply?” Because, again, we have learned how to do reinforcement learning to achieve that better and better and better. Every generation of such systems is more useful because it interprets requests better.

One of Eliezer Yudkowsky's essays, which his acolytes have frequently pushed me towards, is called "The Hidden Complexity of Wishes". In it, he describes an unnecessarily complicated thought experiment that can be summarized in only a few words without real loss of content: "Imagine that you ask an AI to 'get your mother out of a burning building'. It might respond by throwing her out the window, which isn't what you wanted at all! So getting an AI to do what you actually want is really hard!"

One of Eliezer Yudkowsky's persistent claims is that it's nearly impossible to get an to AI understand your requests well enough that it's safe to ask for even a fairly mundane thing like two identical strawberries on a plate. I think the current generation of LLMs already shows that's false. As LLM models have improved, so too has their understanding of common sense context, for example, that you probably don't want to get two identical strawberries if it means killing people to do it.

Why do many of us claim that someone following Eliezer’s claims might resort to terrorism? Because if you tell people over and over again that they and everyone else they know and love will die horribly if the evil AI people aren’t stopped soon, some people might believe you! And if they really, really believe you, what’s the logical next step? Eliezer can say “when I said ‘won’t someone rid me of this meddlesome priest’ I didn’t mean you should rid me of this meddlesome priest!” over and over again…

“This very smart person says something very weird. Should I believe them?” Well, maybe. Some weird things are true! But I’ll point out that very smart people are also prey to delusions far beyond those normal people are capable of. Kurt Gödel, one of the 20th century’s greatest minds, starved himself to death out of paranoia that his food might be tainted. As I often note, John von Neumann was convinced, based on game theory, that a nuclear first strike against the Soviets was necessary for survival.

@AlexVeshev @tszzl As I’ve repeatedly noted, John von Neumann was convinced, based on what he thought was an airtight game theory argument, that a nuclear war was inevitable and a first strike against the Soviets was necessary. @AlexVeshev @tszzl He was the smartest person in history, and yet he was not sane on this matter. Thank goodness he was ignored. There is nothing as dangerous as a smart person in the grip of a totalizing belief system.

Once you believe that something is The Most Important Thing in the World Without Exception, you can easily end up as a human paperclip optimizer (or worse), ignoring all restraint in the name of optimizing for The Most Important Thing.

time.com/6266923/ai-eli… “But it’s The Most Important Thing!” they cry. “How could *anything* be more important? By definition nothing is!” And that, ladies and gentlemen, is the path to hell. You can justify literally any action once you have allowed that mental virus in, and there’s no going back.

For those who don't know: I'm talking here about the core group of bay area "EA" and "rationalist" clique members. They literally constitute a cult. They live in group housing, work with the same people they live with, have apocalyptic beliefs, "debug" each other, etc.

https://twitter.com/perrymetzger/status/1641271896892211200

I'm sure that there was no intent to deliberately create a doomsday cult here, but my understanding is that most cults are not deliberately designed but rather converge onto the same set of characteristics, partially because of human psychology and selection.

The most important characteristic of a cult is the fanatical belief that a particular idea is the most important thing in the world and that it supersedes all other considerations whatsoever. It doesn’t matter if the cult’s beliefs have some connection to reality; indeed, being at least partially true can help a cult in recruiting. It doesn’t matter if the cult is accidental and not deliberate; indeed, genuine belief by the leadership is more convincing.

There are definite dangers in technologies like molecular manufacturing, AI, etc. How will we survive them? Immune systems. Mechanisms designed to catch and stop bad actors. We already have such systems at both the civilizational and individual levels. We have things like armies, police forces, self defense classes, etc., because the world has many people who defect from usual societal norms.

The history of mankind has been the development of more and more general technologies. Language, which can communicate anything, not some particular thing, and writing, which can record and store any language, and the printing press, which can duplicate any writing. The computer, which can carry out any describable process. The telegraph, the telephone, and the internet, increasingly general communications technologies. CNC machines and additive manufacturing, which aren’t built with some specific narrow product in mind.