New preprint w/ @mcxfrank:

How can we ascribe cognitive abilities to language models? We evaluate them! But evals impose challenges separate from the underlying ability of interest. These "task demands" affect LM performance, esp. for smaller models! 1/8 bit.ly/3VMdl2u
A shared goal in psychology and AI is to ascribe cognitive capacities to black-box agents. For example, we might be interested in whether a young child has theory of mind, or whether an LM can distinguish grammatical and ungrammatical sentences. 2/8

New paper with Sammy Floyd, @OlessiaJour, @ev_fedorenko, @LanguageMIT! Non-literal language understanding is an essential part of communication. But what is the role of mentalizing vs. language statistics in pragmatics? & how well do NLP models capture human prag behaviors? 🧵1/6 We explore this through a fine-grained comparison of LMs and humans on 7 pragmatic tasks. Our eval materials are an expert-curated set of multiple choice q's. Each answer option represents different strategies for solving the task (pragmatic, literal, low-level heuristics). 2/6

Excited to share our new preprint (w/ @smallhannahe & @ev_fedorenko): Our results support the idea that language comprehension & production draw on the same knowledge representations, which are stored in the language-selective network. 1/7biorxiv.org/content/10.110… A network of left frontal and temporal brain regions has been implicated in language comprehension & production, but what is the precise role of this ‘language network’ in production? Across 4 fMRI expts, we characterize the response of the lang regions to production demands. 2/7