Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Zayne Sprague Profile picture
Zayne Sprague
@ZayneSprague
Ph.D. student at the University of Texas in Austin. My interest is in NLP, RL and CogSci research focusing on reasoning in AI models. (he/him)
Sep 19 • 8 tweets • 4 min read
To CoT or not to CoT?🤔

300+ experiments with 14 LLMs & systematic meta-analysis of 100+ recent papers

🤯Direct answering is as good as CoT except for math and symbolic reasoning
🤯You don’t need CoT for 95% of MMLU!

CoT mainly helps LLMs track and execute symbolic computation

Image
Image
Image CoT’s effectiveness in the literature is often based on datasets like MATH and GSM8k. Does it work more broadly?

We went through *all* the papers using CoT from ICLR ‘24 and NAACL/EACL ‘24 and collected experiments from over 100 of them.