Thread by @jacobandreas

Speculative (!!!) paper arguing that big LMs can model agency & communicative intent: arxiv.org/abs/2212.01681 (somehow in EMNLP findings). Briefly:

1. LMs do not in general have beliefs or goals. An LM trained on the Internet models a distribution over next tokens *marginalized*

over all the authors who could have produced the context. All large-scale training sets are generated by a mixture of authors w/ mutually incompatible beliefs & goals, and we shouldn't expect a model of the marginal dist. over utts to be coherent.

BUT

2. B/c single training documents are written by individuals who do have specific communicative intentions, and b/c understanding these intentions helps w/ next-word prediction, we should expect LMs to *infer and represent* the latent beliefs/intentions/etc that give rise to a ctx

3. There's lots of evidence that current LMs already do this for coarse notions of "belief", "goal", "intention", etc. The one I find most convincing is the fact that LMs get better at TruthfulQA if you prefix the input w a description of a truthful agent: arxiv.org/abs/2109.07958

So even if an LM isn't a model of a specific agent in general, a good LM can model the communicative behavior of an agent in context, and will do so by building internal representations that have the same causal relation to output that the agent's internal states have.

Obviously current models still make lots of errors when doing this. ChatGPT is especially interesting, since it was clearly trained to model a specific agent rather than the entire distribution, but can be tricked into modeling other agents with carefully designed contexts!

There are likely also architectural limitations that prevent current archs from doing agent-inference-and-simulation in a general & reliable way, not least of which is that it's hard to fully specify beliefs & goals in a few thousand tokens. So even training on text alone,

better models might do a better job of explicitly modeling the latent author of the text the LM is trying to generate, and provide downstream users more transparency and control over this inferred author's internal state.