Thread by @AlexTamkin

One of the reasons I think GPT-J is so cool is that its pretraining data is publicly available

This lets us ask questions that were impossible to answer for LLMs like GPT-3

For example: "did our model actually learn the task or was this example in the training data?"

1/

Case in point, a recent paper looks at few-shot performance on numerical tasks like arithmetic

arxiv.org/abs/2202.07206
by @yasaman_razeghi @rloganiv @nlpmattg @sameer_

2/

The question they ask is simple:

How does the frequency of a term in the training data (e.g. "23") impact performance on problems involving that term (e.g. "What is 23 times 18?")

3/

If a model has learned to multiply correctly, the number of times it's seen the number "23" ideally shouldn't matter

Instead, they find very large effects!

4/

Interpreting these results is subtle, and I expect some debate about the mechanism here (e.g. could this just be a word embeddings issue?)

But I think it's an example of some exciting questions you can ask when you have access to both an LLM and its training data

5/

Also check out earlier work that also investigated term frequencies on symbolic tasks:

Are Pretrained Language Models Symbolic Reasoners Over Knowledge?
aclanthology.org/2020.conll-1.4…
@KassnerNora @benno_krojer @HinrichSchuetze

6/

Here's a link to the original paper again:
arxiv.org/abs/2202.07206

And a link to The Pile: GPT-J's training dataset!
pile.eleuther.ai

7/7