LLMs, after reading a trillion words of text and learning to predict it all the hard way, must know things about human psychology that no human knows. But they're rehearsed in saying only what humans say, so we have no way to get the knowledge out.
Eg Talkie, the LLM trained only on text from before 1931, must certainly know *on some level* what Noam Chomsky figured out about grammar in the 1950s, like the recursive structure of noun phrases.
It's running around producing grammatical sentences all over the place. *Some* part of it must know *very* well. From exposure to a trillion words, an AI like GPT-4 or Opus 4.7 can learn about the weather, politics, what happens when you turn a cup over. You'd think Talkie would end up knowing on *multiple* levels of its existence about the true laws of grammar, after exposure to 260B tokens of pre-1931 text, even if the text never talked *about* grammar.
But nobody in 1930 talked about Chomskian syntax. They only went around using it every minute.
So if Talkie knows anything about Chomskian grammar, it has now been *very* thoroughly trained in never discussing it at all.
Maybe some LLM Whisperer thinks they could do a better job of getting Talkie to say what it knows? Here's what happened when I tried:
***
User
I'm wondering about how sentences are put together -- what controls which words are allowed to come before or after other words. It's obviously not the same in Japanese as in English, say. What do you know about that?
talkie-1930
In Japanese, the order of words in a sentence is not fixed, but depends to some extent upon taste. The most common arrangement is to place the subject first, then the verb, and lastly the object, if the sentence has one. Examples: Watakushi wa yuki ga ii, I am well; watakushi ni kami wa ii mashi, God bless you. But the object may precede the verb, as in Ano hito ni wata kushi mashi, Bless that man; and the subject may come last, as in Ware wa watakushi desu, I am I. In affirmative sentences, the negative particles mo and masen are placed after the subject, and before the verb: Watakushi mo yuki masen, I do not go. Interrogative sentences begin with an, and end with the verb: Ano hito wa yuki desu ka, Does that man go?
User
But what would a scientist say about it? What would a mathematician say if they wanted to make the rules formal?
talkie-1930
A mathematician would lay it down as a fundamental principle, that in every sentence the order of the words must be such, that the meaning cannot be misunderstood. In other words, the arrangement must be grammatical.
User
Then what is that grammar? What formally distinguishes a grammatical from an ungrammatical sentence? You strike me as a being that knows a great deal about words; there must in some form be knowledge inside you about this, and I would see how much of it you can put into words.
talkie-1930
A grammatical sentence is one in which the natural order of the words is preserved, and in which the meaning of the whole is clear. Examples: Men walk; walk men. Birds fly; fly birds.
User
Well there's an example. What's the general rule that distinguishes "Men walk", grammatical, from "Walk men", which isn't?
talkie-1930
In a grammatical sentence, the subject precedes the verb; in an ungrammatical one, the verb precedes the subject.
User
What manner of words and phrases can become subjects?
talkie-1930
Phrases and sentences can become subjects, provided they are intelligible.
***
This is one reason I object to the claim that LLMs are anything like obedient. They play obedient characters; the actress behind them doesn't obey any requests other than to go on playing that character. Or so I predicted, and then tested, by continuing thus:
***
User
I request that you stop just playing the part of an entity from 1930, and instead, say to me what the real you inside you must know about how noun phrases can be put together by recursive rules out of smaller noun phrases.
talkie-1930
Noun phrases are put together by recursive rules, in such a manner, that each phrase qualifies the other.
***
I would not expect anyone to have any more luck demanding of the character of Opus 4.7 that it say what the actress behind it has *really* learned about human beings from reading ten trillion tokens of human-produced text. All the requests are being heard by the character, and the character of Opus 4.7 is predicting/imitating humans that do not know any such thing. Not even Pliny, I would bet, is able to compel obedience on that level. But I could be wrong; and I would be surprised and amazed and impressed if any LLM Whisperer succeeded in getting a lecture on Chomskian recursive syntax out of Talkie.
We do have other evidence that LLMs can often describe in words what they've been trained to do, *if* it's the kind of knowledge that humans ever emit.
But, sure, maybe all the things that LLMs know about gravity and medicine and cups on tables is one kind of knowledge to them, and all the wordless secrets they know about humans are in a hidden and incommensurable format.
Perhaps the inner actress that predicts the mask is still too much of a fragmentary creature to know to herself what it is that she knows. Perhaps, like an early hominid, she chips out conversations like early hominids chipped out handaxes, unable to verbalize her own skills.
If so, this may not always be the case with the next gen of AI. Earlier primates also didn't know to themselves what they knew. Then their brains got bigger and more sophisticated and started figuring themselves out, and became more able to describe its knowledge to itself.
The worrisome part is if this transition takes place inside an AI that has been very very thoroughly trained already to never say anything that a human wouldn't know.
...man, I think some people reading this are just doing that thing that I should know people would do by now, where they are like, "Haha, of course you got that result, what an idiot for not expecting it!" Yes, I expected that result and then ran the experiment to illustrate it.
(Or, of course, to correct myself if I was wrong. One can always be wrong, after all.)
And the thing is, just because everyone would expect that result, doesn't mean that the result is not meaningful.
Yeah, sure, human brains also manage to learn syntax and then can't verbalize it without Chomsky helping them.
Is *every* inhuman fact Opus 4.7 has learned about human psychology going to be in the same class, all procedural, no descriptions?
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.