Thread by @ShriramKMurthi

I hope @gmarceau and @KathiFisler don't find out that I actually named the paper after a very silly British sitcom from the 80s. I expect several Indians will recognize it; but will anyone British? But there's a really important story on research methods behind the paper: 🧵

This work followed-up on our earlier paper, "Measuring the Effectiveness of Error Messages Designed for Novice Programmers". We had done a bunch of work analyzing edits and classifying error output. Seemed like a pretty done deal. But: ↵

We also did a handful of talkaloud studies to see how students really handled errors. Most of it was predictable. But one student said they were confused because they didn't know the WORDS in the error. The words?!? (Native English speaker, too.) ↵

This was the tiniest string to pull on—one remark by one student one time—but we yanked anyway. We realized that there were several mismatches. Profs didn't use the same vocab as the IDE, so even if students knew a concept, they didn't recognize it. Also: ↵

In DrRacket, we had gone to GREAT effort to make the student languages use precise terms: eg, "constructor" vs "selector" vs "predicate". Problem was, students only knew "function". So they were hearing that the frobozicator didn't get a globbeldyglub. WAT. ↵

Still, we had paid a ton of attention, over 15 years, to vocabulary in DrRacket error messages. So we figured it can't be that bad. We decided to run a study to confirm it. But how? Do you see why it's messy? ↵

You can ask students to "define" a term, but they are novices; definitions will be garbled. Anyway, we want to know if they *recognize* a term. So, eg, we gave them programs + error msgs and said "circle all instances of this term in the error in that program". ↵

Our findings were disastrous. In particular, all that careful refinement of DrRacket errors was part of the problem, but not the only part. 3 different institutions, courses, instructors, all miserable. ↵

Another finding was that not only do profs need to align with the IDE, so do libraries. Instead libraries create their own error vocabulary and style, and it clashes with the base language. The result was a STYLE GUIDE for error messages. ↵

ALL this came about because we used QUALITATIVE methods and listened carefully to EVERY word the subjects said. We pulled on the tiniest of strands and discovered a whole dress. Maybe some awesome ML research can do the same…I don't know. ↵

A lot of people have read and incorporated some of the feedback from this work. It's really cool to have had industrial impact. It also shows that to have industrial impact, you don't have to labor in salt mines. Studying beginning students can have direct impact on industry! ↵

Finally, awesome that @gmarceau had three main papers and all three have now won awards.

And today's SPLASH awards also had @mgrnbrg and @ArjunGuha get an artifact award that was presented by @csgordon. Brown PLT represent!!! Like a giant virtual reunion @ the award ceremony. 😃

And for the follow-up to this thread (what came next), see:

https://twitter.com/ShriramKMurthi/status/1451688982761381892