Thread by @ShriramKMurthi

@madeofmistak3 Yes: I've done a ton of work on error messages too (some of which others working on errors have read):
cs.brown.edu/~sk/Publicatio…
cs.brown.edu/~sk/Publicatio…
So that's not really the point I'm making.

@madeofmistak3 Let's read what the slide says: the error message should make it "obvious" how to fix "the" problem. Both of those words are very problematic and suggest an incomplete understanding of the issues or of very sloppy writing or sloganeering. »

@madeofmistak3 Let's first assume there isn't some very advanced AI is running between the error and message generation. Because sometimes it will be spectacularly right and sometimes spectacularly wrong, and in latter case will totally fail this criterion. So assume "traditional" messages. »

@madeofmistak3 Error messages come from _languages_, but errors are made in _programs_. By definition, there's a big semantic gulf between the language and program. Fixes have to be at the level of the program. How can the _language_ make "obvious" the program's problem? »

@madeofmistak3 This also assumes that there is "the" problem. Many times an error is the result if an *inconsistency* (trivial example: f takes two args and is given three; not clear whether caller or callee is to blame). In our research we found …»

@madeofmistak3 … error messages often blamed one party rather than both, which resulted in people fixing the wrong thing, thinking the omniscient computer had told them where to fix. By making things point to inconsistency, we made things less "obvious" in return for not misleading users. »

@madeofmistak3 But let me give you a great example of how subtle errors can be. In one of our studies, we found that in week ~4/5 of semester, students suddenly had spate of "arity mismatch" errors. Had had them early on, then they largely disappeared, then suddenly reappeared. WTF?!? »

@madeofmistak3 A detailed analysis (good human factors/data science work => you have to READ THE DATA, not just run some stupid algorithm) revealed it was all on a specific constructor. Why were they screwing up just that one constructor? »

@madeofmistak3 Turns out it was a lab w/ two problems, and they were screwing it up mostly in the second problem (most students were doing second after first). Plot thickens. Then we read the lab problem specs carefully and found it. »

@madeofmistak3 The previous problem had defined the same structure with different arity. Wasn't this, but think prob 1 says "point has x, y" and prob 2 says "point has x, y, z". Proactive interference: they suddenly started making arity errors. »

@madeofmistak3 The "obvious" fix is "Please forget what you know about points from previous problem and re-learn the new def, since you're going to stumble on this several times in this problem". But the *real* fix is … »

@madeofmistak3 "Please check in with your prof whether they really meant to confuse you like this. If so, fine, otherwise tell them to rename the structs." Prof did NOT intend this and many facepalms ensued. »

@madeofmistak3 Errors live in a very complex ecosystem. As a programmer, course I'd love what the slide asks for. As a researcher and language designer and curriculum author, I'd be terrified of anything that makes such claims. Even as someone who's spend 8 hard years now on better msgs. •

@madeofmistak3 (PS: 8 years of formal research, 20+ years of actually creating error messages and seeing them used in practice and refining them.) ••

@madeofmistak3 For people wondering how subtle error msg situations can be, here are many in @PyretLang [probably several more that we forgot/failed to tag]. Look at how many different things they interact with! [And have I said how much I hate, Hate, HATE infix syntax?]
github.com/brownplt/pyret…