Automating AI research is exciting! But can LLMs actually produce novel, expert-level research ideas?

After a year-long study, we obtained the first statistically significant conclusion: LLM-generated ideas are more novel than ideas written by expert human researchers. In our new paper:

We recruited 49 expert NLP researchers to write novel ideas on 7 NLP topics.

We built an LLM agent to generate research ideas on the same 7 topics.

After that, we recruited 79 experts to blindly review all the human and LLM ideas.

2/ arxiv.org/abs/2409.04109

One powerful / scary application of LLMs is using them to persuade humans, for good or bad.

I read through recent empirical human studies on LLM persuasion, and here’s a thread summarizing my favorite ones:

(google doc version: )

1/ 🧵docs.google.com/document/d/1il… I’ll summarize each paper along a few key dimensions:

- Persuasion topics
- Interaction format
- Measurement of persuasiveness
- Main findings

Let’s start with a few studies on measuring the persuasiveness of LLMs.

2/