📢The problem in model alignment no one talks about — the need for preference data, which costs $$$ and time!

Enter Kahneman-Tversky Optimization (KTO), which matches or exceeds DPO without paired preferences.

And with it, the largest-ever suite of feedback-aligned LLMs. 🧵 But first, what makes alignment work? Among methods that directly optimize preferences, the majority of gains <30B come from SFT.

Even a dummy one-step PPO that uses +1/-1 rewards works very well.

DPO is uniquely good at the 30B scale, however. 2/

📢 Models like #ChatGPT are trained on tons of human feedback. But collecting this costs $$$!

That's why we're releasing the Stanford Human Preferences Dataset (🚢SHP), a collection of 385K *naturally occurring* *collective* human preferences over text.
huggingface.co/datasets/stanf… Given some context and two possible responses, SHP preferences reflect the helpfulness of one response over another.

The preferences are over responses to questions/instructions in 18 domains, from cooking to legal advice, drawn from Reddit.

Is there a connection between Shapley Values and attention-based explanations in NLP?

Yes! Our #ACL2021NLP paper proves that **attention flows** can be Shapley Value explanations, but regular attention and leave-one-out cannot.



w/ @jurafsky @stanfordnlp arxiv.org/abs/2105.14652
Shapley Values are a solution to the credit assignment problem in cooperative games -- if 10 people work together to win some reward, how can it be equitably distributed?

For this reason, they've become a popular kind of explanation in ML. 2/

There's been some confusion over what Microsoft's "exclusive license" really means here.

While I can't speak for OpenAI, exclusive licenses generally grant exclusivity *within some specific context*. So no, Microsoft won't be the only one able to use GPT3. That said ...

https://twitter.com/kevin_scott/status/1308438898553638912

My guess is that only MS will have access to the underlying model, while everyone will have to go through the API and be at the whims of whatever terms are set by OpenAI.

Inspired by @yoavgo 's poll, I looked at the views for papers in three tracks -- Ethics, Summarization, and Theme (69 papers in total).

The median views per paper was 104.

In these three tracks, the most-viewed papers at time of writing are ...

https://twitter.com/yoavgo/status/1282087972553338880

1. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data by @emilymbender and @alkoller (961 views)

2. How Can We Accelerate Progress Towards Human-like Linguistic Generalization? by @tallinzen (410 views)

Is your NLP classifier actually (un)biased? Or is your diagnosis based on too little data?

It might be the latter!

In my #ACL2020 paper, I discuss why we need bigger datasets for conclusively identifying classification bias in NLP.

arxiv.org/abs/2004.12332 1/ Background: Large NLP datasets don't come with annotations for protected attributes (e.g., gender). To test for classification bias, one typically annotates a small sample of data (typically < 5K). WinoBias and WinoGender are great examples of these bias-specific datasets. 2/

How contextual are contextualized word representations? In a long-overdue blog post, I discuss my EMNLP paper on how ELMo, BERT, and GPT-2 contextualize words — how they’re alike, and how they’re different.

blog: kawine.github.io/blog/nlp/2020/…
paper: aclweb.org/anthology/D19-… Key findings:

1. In all layers of BERT, ELMo, and GPT-2, the representations of all words are anisotropic: they occupy a narrow cone in the embedding space instead of being distributed throughout.

What causes bias in word embedding associations? Biased training data, the embedding model, or just noise?

In a long-overdue blog post, I discuss our ACL paper on understanding undesirable associations. #nlproc

blog: kawine.github.io/blog/nlp/2019/…
paper: aclweb.org/anthology/P19-…

1/5 Key Takeaways:

The vast majority of words are not, on average, any more gendered in a word2vec embedding space than they are in the training corpus. Exceptions are words that are gender-stereotyped (e.g., ‘nurse’) or gender-specific by definition (e.g., ‘queen’). 2/5

When and why does king - man + woman = queen? In my #ACL2019 paper with @DavidDuvenaud and Graeme Hirst, we explain what conditions need to be satisfied by a training corpus for word analogies to hold in a GloVe or skipgram embedding space. 1/4

blog: bit.ly/2X18QEd paper: arxiv.org/abs/1810.04882

In turn, our theory provides
1. An information theoretic interpretation of Euclidean distance in skipgram and GloVe embedding spaces.
2. Novel justification for the surprising effectiveness of using addition to compose word vectors. 2/4