Thread by @random_walker

Great example of why you shouldn't read too much into analogies like "bullshit generator" and "blurry JPEG of the web". The best way to predict the next move in a sequence of chess moves is… to build an internal model of chess rules and strategy, which Sydney seems to have done.

This also illustrates an important LLM limitation: you can't be sure from its behavior that it has learned the rules correctly. There may be subtle divergences that show up only once in a million games. (Fun fact: chess does have move-limit rules that trigger exceedingly rarely.)

This may not matter much for chess playing, but in consequential applications such as robotics, 1-in-a-million errors might not be acceptable.

It's not surprising that chess-playing ability emerges in sufficiently large-scale LLMs (we've seen emergent abilities enough times). What's very surprising is that Bing's LLM is apparently already at that scale.

ChatGPT couldn't play chess at all — couldn't consistently make legal moves, couldn't solve mate in 1 in a K+Q vs K position. Sydney, on the other hand, has not only learnt the rules, but can play reasonably good chess! Far better than a human who has just learned the rules.

Of course, this shouldn't be surprising — when learning chess through next-move prediction, learning the rules isn't actually a prerequisite for learning strategy, and it's learning both together. Still surprising to actually see it in action.

Yes, an extraordinary claim, but I don't think it's possible to play passable chess without building an internal representation. Chess moves are far too sensitive to the details. And if you can model all those details, that *is* an internal representation.

This is, of course, a big fault line in the debate about LLMs — one camp says it's all just statistics, regardless of the complexity of the behavior. FWIW, I'm in the internal representation camp. The good news is it's empirically testable! Early work: thegradient.pub/othello/

Interesting hypothesis but seems implausible to me. Just as LLMs learn writing style and grammar by learning to predict words, it seems far more likely that they learn chess by predicting moves in the over 1 billion (!) games available online.

Internal representation or not, LLMs do no chess calculation (game tree search). So my guess is that no matter the scale, they can't come close to chess engine performance or even expert human performance. I'd love to be proven wrong, of course.

But imagine that the bot can recognize that it is looking at a description of a chess position and invoke Stockfish (a chess engine) to figure out how to continue. I find this research direction — augmented language models — tremendously exciting. arxiv.org/abs/2302.07842

Question in the replies: what does internal representation mean? The main criterion is whether it tracks the board position as it sees the moves. This already requires decoding the notation and learning how each chess piece moves. Same criterion used here: thegradient.pub/othello/

The alternative would be to learn a function that maps every possible *history* of moves to the set of legal moves in the resulting position, without first computing the position. That is so implausible that it seems obvious that the model must keep track of the board.

Similarly, we can ask if it has learned strategic concepts (material value, king safety, ...) that would allow it to efficiently generate good moves, versus something brute force. Again, this is empirically testable — and has been confirmed in AlphaZero! pnas.org/doi/10.1073/pn…

To state the obvious, all of my claims about Bing in this thread are guesses based on circumstantial evidence. This is why transparency is so important (what's in the training set?) as are open-source models that we can probe and understand.

As a reminder, the blackboxness of neural networks is vastly exaggerated. We have fantastic tools to reverse engineer them. The barriers are cultural (building things is seen as cooler than understanding) and political (funding for companies vs for research on societal impact.)

BTW, for folks seeing this thread who don't follow me, this is coming from someone who helped spread the bullshit generator analogy. I'm not against analogies! They're somewhat helpful, but lack nuance. We need way more research on LLM abilities & limits. aisnakeoil.substack.com/p/chatgpt-is-a…

Amazing — a paper from two years ago on the exact question we've been fighting about in this thread! My favorite thing about Twitter.