Thread by @ShriramKMurthi

1/ Recently got access to Google's Illuminate, which AI-generates a ~6min "podcast"-format summary of a research paper. To test it, I gave it 5 of my own recent papers. This thread summarizes the evaluation: ↵

2/ I gave it my papers because I could most definitively evaluate how well it's doing. I gave it recent papers so it's likely there's very little data about them on the Web, simulating the "this paper just came out, help me grok it" mode. ↵

3/ I also chose papers across a variety of areas. The topics range across formal/theory, programming languages, systems, human-factors, software engineering, and education (and most a mix of these). This is to see how it fares across CS topics. ↵

4/ I picked 5 because you can do 5/day. After generating them, I sent them to my *co-authors* to see how *they* felt about the generated summary. Their scores are also below. Before giving scores, let me say two nice things about Illuminate: ↵

5/ a. The two voices sound very natural. The conversation actually flows (sort of, it's slightly artificial, but have you heard some of the *podcasts* out there?). That actually may make it sound more informed than it might be! ↵

6/ b. People have asked, "why the q&a format?" My answer: When you're reading, you want section headers or other markers of transition. That's hard to do in audio. The switch to another voice asking a question essentially serves as a human-friendly section header. ↵

7/ Now, one thing I didn't do well is pin down exactly what the grading criterion would be. My own view was: «Imagine that this is the summary of the paper presented by a student taking a papers-based grad seminar. What score would I give that student?» ↵

8/ But some people viewed it as, "Imagine an *undergrad* were asked to summarize the paper" (implicitly, an undergrad w/out much research experience, not the ones that function like grad students). That led to some grade "inflation" (due to different expectations). ↵

9/ Basta! Enough! This is like a recipe that first spends three paragraphs on the author's childhood and travels. I know why you're really here: you just want to see the grades! ↵

10/ Common issues: It makes some mistakes, but most of all it misses a lot of things, including some of the most important ideas. Often the specific parts were (effectively) about context and background, the parts that may be in LLMs, not so much about THIS paper. ↵

11/ It is just like a novice researcher: it gets a general sense of what's going on, doesn't always know what to focus on, sometimes does a fairly good idea of the gist (especially for "shallower" papers), but routinely misses some or ALL of what makes THIS paper valuable. ↵

12/ Overall, anything that can help more people learn more is good. But there are currently far better ways to achieve that. (That's why we blog!
) Illuminate will improve! But I regret to say that for now, you're going to have to actually read papers. (-:blog.brownplt.org