Kawin Ethayarajh Profile picture
Jun 23, 2020 8 tweets 4 min read Read on X
Is your NLP classifier actually (un)biased? Or is your diagnosis based on too little data?

It might be the latter!

In my #ACL2020 paper, I discuss why we need bigger datasets for conclusively identifying classification bias in NLP.

arxiv.org/abs/2004.12332 1/
Background: Large NLP datasets don't come with annotations for protected attributes (e.g., gender). To test for classification bias, one typically annotates a small sample of data (typically < 5K). WinoBias and WinoGender are great examples of these bias-specific datasets. 2/
Intuitively, the less data we annotate, the less certain we are that our estimate is close to the true bias. But how can we quantify this uncertainty? 3/
I suggest framing classification bias as the difference in expected cost across the protected and unprotected groups. Difference fairness measures (e.g., equal opportunity) have different cost functions. 4/
By treating an example's contribution to the overall bias as a random variable, we can apply Bernstein bounds to come up with a confidence interval for the overall bias.

This proposed approach is called Bernstein-bounded unfairness (BBU for short). 5/
BBU suggests that bias-specific datasets used in NLP are often too small to conclusively identify bias.

For example, say a coref resolution system is 5% better on gender-stereotypical inputs. To claim bias with 95% confidence, we'd need a dataset 3.8x bigger than WinoBias! 6/
Takeaways / Future Work:

#1 The NLP community needs much larger bias-specific datasets so that we can make claims about the presence / absence of bias with high confidence.

#2 Tighter bounds on bias estimates can help us make more confident claims with less data.

7/
This work was done at @stanfordnlp and @StanfordAILab. Many thanks to @sulin_blodgett, @haldaume3, @s010n, and @hannawallach for including this in their recent lit review! arxiv.org/abs/2005.14050

8/

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Kawin Ethayarajh

Kawin Ethayarajh Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @ethayarajh

Dec 7, 2023
📢The problem in model alignment no one talks about — the need for preference data, which costs $$$ and time!

Enter Kahneman-Tversky Optimization (KTO), which matches or exceeds DPO without paired preferences.

And with it, the largest-ever suite of feedback-aligned LLMs. 🧵 Image
But first, what makes alignment work? Among methods that directly optimize preferences, the majority of gains <30B come from SFT.

Even a dummy one-step PPO that uses +1/-1 rewards works very well.

DPO is uniquely good at the 30B scale, however. 2/ Image
But *why* do they work?

We find that alignment methods impute a utility function to humans.

These imputed functions have many qualities of those empirically derived by Kahneman & Tversky in their Nobel Prize-winning work on how humans make decisions about uncertain outcomes. 3/
Read 13 tweets
Feb 22, 2023
📢 Models like #ChatGPT are trained on tons of human feedback. But collecting this costs $$$!

That's why we're releasing the Stanford Human Preferences Dataset (🚢SHP), a collection of 385K *naturally occurring* *collective* human preferences over text.
huggingface.co/datasets/stanf…
Given some context and two possible responses, SHP preferences reflect the helpfulness of one response over another.

The preferences are over responses to questions/instructions in 18 domains, from cooking to legal advice, drawn from Reddit.
They were inferred from the simple observation that if comment A was written after B but has a higher score despite getting less visibility, then ostensibly A > B.

If A was written before B, then we can't conclude this -- the higher score could have come from more visibility!
Read 10 tweets
Jun 5, 2021
Is there a connection between Shapley Values and attention-based explanations in NLP?

Yes! Our #ACL2021NLP paper proves that **attention flows** can be Shapley Value explanations, but regular attention and leave-one-out cannot.



w/ @jurafsky @stanfordnlp arxiv.org/abs/2105.14652
Image
Shapley Values are a solution to the credit assignment problem in cooperative games -- if 10 people work together to win some reward, how can it be equitably distributed?

For this reason, they've become a popular kind of explanation in ML. 2/
Shapley Values have been used to explain the importance of individual features, embeddings, and neurons.

@GhorbaniAmirata and @james_y_zou have even used them to value training data points.

In NLP though, attention-based explanations and leave-one-out still predominate. 3/
Read 8 tweets
Sep 22, 2020
There's been some confusion over what Microsoft's "exclusive license" really means here.

While I can't speak for OpenAI, exclusive licenses generally grant exclusivity *within some specific context*. So no, Microsoft won't be the only one able to use GPT3. That said ...
My guess is that only MS will have access to the underlying model, while everyone will have to go through the API and be at the whims of whatever terms are set by OpenAI.
This is big -- if you build a product on top of GPT3, your ability to scale will depend on OpenAI's willingness to increase your throughput, which in turn will depend on the terms of their agreement with MS. Not a great situation to be in if you're directly competing with MS.
Read 6 tweets
Jul 12, 2020
Inspired by @yoavgo 's poll, I looked at the views for papers in three tracks -- Ethics, Summarization, and Theme (69 papers in total).

The median views per paper was 104.

In these three tracks, the most-viewed papers at time of writing are ... Image
1. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data by @emilymbender and @alkoller (961 views)

2. How Can We Accelerate Progress Towards Human-like Linguistic Generalization? by @tallinzen (410 views)
3. The Unstoppable Rise of Computational Linguistics in Deep Learning by @JamieBHenderson (356 views)

4. (Re)construing Meaning in NLP by @Sean_Trott @TorrentTiago @nancy_c_chang @complingy (291 views)
Read 6 tweets
Feb 7, 2020
How contextual are contextualized word representations? In a long-overdue blog post, I discuss my EMNLP paper on how ELMo, BERT, and GPT-2 contextualize words — how they’re alike, and how they’re different.

blog: kawine.github.io/blog/nlp/2020/…
paper: aclweb.org/anthology/D19-… Image
Key findings:

1. In all layers of BERT, ELMo, and GPT-2, the representations of all words are anisotropic: they occupy a narrow cone in the embedding space instead of being distributed throughout.
2. In BERT, ELMo, and GPT-2, upper layers produce more context-specific representations than lower layers; however, the models contextualize words very differently from one another.
Read 7 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(