Thread by @timconverse

Why doesn’t a good negative history of attempts to get to Artificial General Intelligence (AGI) exist? What is it about AGI that makes such a treatment not only hard to find, but hard to create?

In this thread, I’ll propose reasons for
1) hard to find, and
2) hard to create

Background: David Chapman (@meaningness) tweeted that there have been around 8 attempts at AGI, and that “someone” should document each attempt and reason for failure (I took the scare quotes to mean that maybe he himself would do it, maybe he wouldn’t).

https://twitter.com/Meaningness/status/1362813795303780354

*PART 1* (Hard to find): it’s true that if you look at AI textbooks and conferences, all you will find with regard to AGI is the positive residue of each AGI attempt. The attempts are easy to find, and the negative verdicts are absent, even though none have succeeded (yet).

I don’t know which 8 Chapman had in his head, but I bet that anyone with knowledge of AI could name 8, and they would mostly overlap with his. If they differ, it’s going to be about granularity (is ML 1 attempt or 4? Do 3/4 of the attempts get lumped under "logic" Etc.)

Here are 13 “attempts” that come to mind for me, all well-hyped for AGI at the time:
1) the logicist programme (McCarthy)
2) neural nets
3) natural language parsing
4) knowledge bases and Cyc
5) reinforcement learning
6) rule-based expert systems

7) classical planning and problem-solving
8) Problem-space search (SOAR)
9) Schankian scruffy AI, case-based reasoning
10) genetic algorithms
11) robotics (including Brooksian)
12) *deep* neural nets [worth its own category]
13) whole-brain emulation

What a good Popperian might hope for is:
1) Hypothesis: Technique or area of study X is the route to general AI!
[a few years of hard work]
2) Yeah nope! Didn’t work. And here’s why: [well-documented disconfirmation]
OK, on to the next!

Instead, for sociological reasons, it works more like this:
(Phase 1) Researcher: “Technique X is the route to AGI! And [implicitly], even if it’s not, we’ll have fun and learn a lot and extend the boundaries of what’s known about AI, and produce useful programs! Give us money!”

[years of work ensue]
(Phase 2) lots of publications, the creation of a named subfield of AI, lots of code that does things that in some sense couldn’t be done by computers before, which may or may not be useful.

If they’re useful, new businesses get formed. It’s even true in many cases that the boundaries of what’s known are extended, in that some tasks turn out to be automatable and others not so much, and no one knew going in which would be which.

(Phase 3) Over time, the subfield may be viewed as less important – funding dries up, fewer papers are written, there’s the sense that the low-hanging fruit has been picked.

With regard to the original claim of being the route to AGI, there is no incentive for anyone to do a nail-in-the-coffin AGI postmortem. Certainly not the original researcher who originally proposed it – after all they now are a well-known founder of an AI subfield!

Also the positive residue is often useful from an engineering point of view. Lots of code gets written to try to solve real problems, and some of it is useful.

Any number of general “cognitive architectures” that have been proposed got slowly demoted into being mere software frameworks, but people still found them useful for writing some AI-ish programs more quickly.

So part of the “hard to find” explanation is around sociology and incentives – there’s little motivation to write a post-mortem. But also the eng goals shade into the philosophical goals and real-AGI goals, and there’s no clear point at which to call the whole thing a failure.

The AGI dream that fuels a lot of AI research plays a similar role to the SETI dream that fuels a lot of astrophysical work. When the work is good, it extends the field and techniques get better. But it wouldn’t get the same money and press if it wasn’t wrapped around the dream.

What kinds of negative results or disconfirmations would you want to have for AI?

Examples of notable negative results from other fields which have put boundaries around their ambitions:

Hilbert: we will automate Math and prove all the theorems!
Godel: No you won’t (proof)

Quantum-mechanics hidden-variable theorist: We’ll make a quantum interpretation that’s not obviously crazy!
Bell’s theorem: Pick your favorite flavor of crazy.

Voting idealist: We’ll make a voting scheme that’s totally fair!
Arrow’s theorem: Pick your unfairness

Distributed systems architect: Our system will be up-to-date, and responsive, and tolerant of network failures!
Brewer and others: Choose your failure mode

Complexity theorist: We’ll prove P!=NP.
[Razborov and others]: Maybe you will, and maybe you won't, but you’re not going to do it this way, or that way, or this other way.

It would be nice if AI and AGI had some crisp negative results like this.

*PART II* (Hard to create) Now on to the question of why negative verdicts on particular AGI attempts are hard to write up. There are at least 4 reasons:

1) AGI is clearly possible
2) AGI isn’t crisply defined (and neither are the AGI proposals)
3) Diminishing returns are in the eye of the beholder
4) Hybrids and bricolage: any AGI attempt might actually be part of the solution

1) AGI is clearly possible. This is a claim that many would want to debate forever, but I don’t want to. If you’re a materialist, then you believe that GI’s like us are a product of physical processes.

There shouldn’t be anything stopping us from building AGI’s on top of similar physical processes – at least nothing at the level of physics. (Well, no-cloning theorems from QM, possibly, for brain-emulations).

Now, maybe we’re not smart enough to build it. But AGI is possible, and the fact of AGI’s possibility also means that we won’t get to a clean impossibility proof a la the Incompleteness Theorem. (I've run into my 25-tweet thread limit - see @timconverse for the rest.)

The possibility of AGI (by any technique) also makes it harder to prove that a particular technique can’t possibly be implicated in the eventual solution.

2) AGI isn’t crisply defined. You have operational evaluations like the Turing test, or (better) embodied analogs like the make-me-some-coffee test, but other than that there’s no real characterization of what such a system has toim be like internally or do.

For some of the examples above (quantum mechanics, voting, distributed systems) you can boil the requirements down to small set of mathematically precise requirements, and then equally precisely prove that they can’t all hold at the same time.

But requirements for AGI are more like walking and chewing gum at the same time, and being able to tell a joke, even if badly, and being able to learn some math reasoning, and being able to keep up your end of a conversation (even though no human actually does perfectly at this).

This vaguely-posed set of abilities is hard to prove negative theorems about.

3) Diminishing returns are in the eye of the beholder. For subfields of AI that are metrics-driven, notions of progress in the subfield can look crazy from the outside.

You can have conversations with zealots with the following flavor:
“How accurate do you need to be?”
“99.99%!"
“How accurate are you now?”
“88.88%! But we’re making so much progress! 3 years ago it was 80%, 2 years ago it was 88%, and last year was 88.8%. The sky’s the limit!

Less quantifiable progress is even worse in this respect. Let’s say that you believe in the original Cyc hypothesis – that our systems fail to be AGI because they don’t know enough commonsense facts, and so we just need to write down enough facts (in a suitable logical formalism)

From the inside it looks like gangbuster progress – more and more facts written down all the time! from the outside it looks like they are spinning their wheels and not getting closer to the original goal. This is halting-problem-esque - How do you know when to declare failure?

It’s worth noting that one of most famous “proven” limits on a potential AI was also essentially wrong.

Early neural-net researcher: Our neural nets will learn all the things!
Minsky and Papert (1969): They can’t even learn exclusive-or, provably!

But this only applied to shallow neural networks, not deep ones, and although it was mathematically correct, it was also completely incorrect about whether the field was promising or not.

It is possible to make trenchant critiques of AI subfields about why they are wrongheaded, aren’t even framing their own problems well, and therefore are unlikely to make much progress even on their own terms.

These critiques often come from outside the subfield as well as outside of AI. IMHO some of them have been essentially right (Dreyfus on symbolic AI, Chapman and Agre on classical planning), and some have just been philosophically confused (Searle).

But with regard to particular AGI candidate subfields it will be hard to agree post-mortem on a consensus cause of death when some of the participants think that their candidate is still fogging up the mirror.

I also think critics in these debates get mileage out of a combo of A) stating the supposed AGI claim in fullest generality, and B) construing the boundaries of the proposed technology as tightly as possible.

Whereas proponents often do the opposite – they hedge (A) (this is a step on the road to an eventual….) and (B) (this, combined with other techniques, shows promise toward an eventual ….).

4) The modularity of real cognitive systems suggests there isn’t a single algorithm that underlies human GI. So maybe the answer will look like a synthesis of AGI approaches to date, + others that we haven't thought of. Or maybe less of a synthesis, and more of a bricolage.

Maybe the AGI robot of the future (the one that passes the Turing test as well as the coffee test) will have something like 2021-era deep neural nets driving its perceptual capabilities.

Maybe the deep-NN pieces will include multimodal deep-NN representations doing something like sensor fusion, and 2021-era DL attention mechanisms giving focus to important lower-level info, and making distant and serendipitous connections.

Maybe those neural nets will be combined with a reinforcement-learning regime along the lines of AlphaGo. Maybe the lower-level motor routines and reflexes will be reminiscent of Brooksian robotics.

Maybe in a part of it there will be something like logical rules represented in terms of NN representations, and something that looks a bit like a rules engine that can draw conclusions from arbitrary rule combinations.

Maybe part of it that will look like like a database of assertions that can be fed to the rules engine.

Maybe there will be a Schankian episodic memory into which notable experiences are encoded, and then retrieved by matching high-level thematic structure (aka “remindings”). Maybe those representations and their matching will owe something to 2021-era graph neural networks.

Or maybe not.

Maybe there will be no AGI solution that we are collectively able to come up with, or maybe the solution that does emerge is based on completely new concepts, throwing all of the above away.

But If an eventual AGI emerged that incorporated most of the above ideas, then all of the contributors might feel justified in saying “See! I *told* you we were on the road to AGI!” and they would feel that the reports of the death of their AGI proposal had been much-exaggerated.