Thread by @spiantado

Everyone seems to think it's absurd that large language models (or something similar) could show anything like human intelligence and meaning. But it doesn’t seem so crazy to me. Here's a dissenting 🧵 from cognitive science.

The news, to start, is that this week software engineer @cajundiscordian was placed on leave for violating Google's confidentiality policies, after publicly claiming that a language model was "sentient"
nytimes.com/2022/06/12/tec…

Lemoine has clarified that his claim about the model’s sentience was based on “religious beliefs.” Still, his conversation with the model is really worth reading:
cajundiscordian.medium.com/is-lamda-senti…

The response from the field has been pretty direct -- "Nonsense on Stilts" says @GaryMarcus

https://twitter.com/GaryMarcus/status/1536087306062352384

Gary's short piece cuts to the core of the issues. The most important is over-eagerness to attribute intelligence. This experiment from the 1940s illustrates it: people perceive beliefs, emotions, intentions, even when shown only moving shapes.

But, beyond that warning, I'm not sure I agree with much. First, it's just not true that systems which only do "pattern matching" are necessarily cognitively impoverished.

In fact, we've known since the earliest days of computing that just pattern matching (e.g. systems of rules that match patterns and rewrite) is capable of *arbitrary* computation.
en.wikipedia.org/wiki/Post_cano…

So even a model that learns really well over "pattern matching" rules is potentially learning over the space of all computations.

(And that, btw, is a pretty good guess for what human learners do cell.com/trends/cogniti… )

This means that a smart "pattern matching" model might, in principle, acquire any computational structure seen in the history of cognitive science and neuroscience.

In other words, what matters is NOT whether the system uses “pattern matching” or is a “spreadsheet.” What matters is what computations it can actually learn and encode. And that’s far from obvious for these language models, which carry high-dimensional state forward across time.

Many have also doubted that large language models can acquire real meaning. The view is probably clearest in this fantastic paper by @emilymbender and @alkoller
openreview.net/pdf?id=GKTvAcb…

Bender and Koller use "meaning" to be a linkage between language and anything else (typically stuff in the world). Their "octopus test" shows how knowing patterns in language won't necessarily let you generalize to the world.

I guess I lean agnostic on "meaning" because there ARE cognitive theories of meaning that seem accessible to large language models--and they happen to be some of the most compelling ones.

One is that meaning is determined, at least in part, by the relationships between concepts as well as the role they play in a bigger conceptual theory.

To use Ned Block's example, "f=ma" in physics isn't really a definition of force, nor is it a definition of mass, or acceleration. It sorta defines all three. You can’t understand any one of them without the others.

The internal states of large language models might approximate meanings in this way. In fact, their success in semantic tasks suggests that they probably do -- and if so, what they have might be pretty similar to people (minus physical grounding).

To be sure (as in the octopus example) conceptual roles don't capture *everything* we know. But they do capture *something*. And there are even examples of abstract concepts (e.g. "prime numbers") where that something is almost everything.

It’s also hard to imagine how large language models could generate language or encode semantic information without at least some pieces of conceptual role (maybe real meaning!) being there. All learned from language.
nature.com/articles/s4156…

Conceptual roles are probably what allows us, ourselves, to talk about family members we've never met (or atoms or dinosaurs or multiverses). Even if we know about them just from hearing other people talk.

For the big claim.... Not a popular view, but there's some case that consciousness is not that interesting of a property for a system to possess.

Some model happens to have representations of its own representations, and representations of its representations of representations (some kind of fancy fixed point combinator?)....

... and so what!

Why care if it does, why care if it doesn't.