Thread by @rao2z

🧵Been reading several recent arXiv entries claiming planning capabilities of #LLM's. This area is so full of anthropomorphisms--"Chain of Thought Prompting", "Inner Monolog"--that it cries out for a cleansing read of Drew's #AI meets Natural Stupidity 1/

https://twitter.com/rao2z/status/889509356084928518?s=20

One popular line claims that while LLM's may give wrong plans, they can improve with right prompting (which, in one case, is claimed to even induce "inner monolog" all West World host-like).

The prompts in the appendices however seem to suggest a Clever Hans effect in action 2/

Clever Hans, btw, is the ChatHorseGPT of its time--that showed very un-horselike arithmetic prowess by stomping its foot the right number of times, as long as the questioner was in front and knew the answers. Crucially, no fraudulent intent was needed. 3/
en.wikipedia.org/wiki/Clever_Ha…

Coming back to LLMs, most successful examples of iterative prompting in action, not surprisingly, involve cases where the questioner does know (or can validate using an external simulator)--the LLM's answer. 4/

https://twitter.com/rao2z/status/1645262542556762112?s=20

It may be fine to give credit to LLMs for finally getting the desired result if the solution criterion is amorphous (e.g. the kind of essay/email you want). Not so much for planning and reasoning where there are well defined notions of correctness! 5/

https://twitter.com/rao2z/status/1624881790212251649?s=20

It doesn't help that underneath the hood, many prompting strategies tend to be very problem specific--thus allowing for unwittingly giving useful debugging hints to the LLM.

If you are giving debugging hints, you are an external solver for the LLM! 6/

https://twitter.com/rao2z/status/1624962256969822208?s=20

I am *not saying* that the people advocating iterative prompting for planning/reasoning are knowingly cheating. Like the owner of Clever Hans, the prompters of GPT3 don't necessarily have to set out to give it sneaky help--and yet give it anyway. 7/

A related issue is the confusion in some of the papers about what makes planning hard. For example, the "Inner Monolog" paper says it uses a "3 block stacking" benchmark, but qualify that they "pre-place" two of the blocks in the correct configuration already. 8/

Well, the combinatoric search aspect of planning comes because of the *subgoal interactions*! If you have three blocks to stack, and put two of them in the correct configuration already, there are no interactions to begin with.. 9/

No wonder then that I have to give deliberate debugging help to get the LLMs--be it ChatGPT (see 5 above) or Bard (which I finally played with today👇 ) to finally blurt out the correct solution for the Sussman Anomaly.. making me wonder who *actually* solved the problem.. 10/

..and I already wrote elsewhere, generating general blocks world plans directly is still well beyond the reach even of GPT4 (@TheEconomist claims notwithstanding..) 11/

https://twitter.com/rao2z/status/1643463201462579200

(This thread is as much for my students, who are sometimes mad at me that I keep poking holes in their attempts to join the gold rush of chain-of-thought gangs..) 12/