Thread by @_jennhu

New paper with Sammy Floyd, @OlessiaJour, @ev_fedorenko, @LanguageMIT! Non-literal language understanding is an essential part of communication. But what is the role of mentalizing vs. language statistics in pragmatics? & how well do NLP models capture human prag behaviors? 🧵1/6

We explore this through a fine-grained comparison of LMs and humans on 7 pragmatic tasks. Our eval materials are an expert-curated set of multiple choice q's. Each answer option represents different strategies for solving the task (pragmatic, literal, low-level heuristics). 2/6

So, how do models do? The larger models achieve high accuracy, and also make similar error patterns as humans: within incorrect responses, these models tend to select the literal interpretation of an utterance over distractors based on heuristics such as lexical similarity. 3/6

We also found that models use similar linguistic cues as humans to solve the tasks. For many tasks, humans and models align on which items are difficult. We also removed the context story from the items, and found that models and humans degrade across tasks in similar ways. 4/6

Our results suggest that even paradigmatic pragmatic phenomena (e.g., polite deceits) could potentially be solved w/o explicit representations of other agents’ mental states, and that artificial models can be used to gain mechanistic insights into human pragmatic processing. 5/6

Thanks to my amazing co-authors for a very fun collaboration! Paper here: arxiv.org/abs/2212.06801 6/6