Machine learning for decision making often results in discriminatory systems. The reason for this isn't specific to ML. It applies to many quantitative models—models that start by assuming a fixed, exogenously specified distribution of decision subjects' characteristics. 🧵
But the distribution of people's attributes (skills, risk levels, preferences…) isn't fixed. In fact, it is usually produced largely by the past effects of the types of decision systems that these models are used to justify. So the logic underlying these models is tautological.
Unless a model can endogenize the risk distribution, it has zero normative content. At best it has some descriptive value to explain observed differences. But these lazy models are so often used to make policy and thus become excellent tools for reinforcing the status quo.
But math doesn't have to be used this way. We can learn to build more sophisticated models, use a diversity of models instead of pretending there's one right answer, and learn to recognize situations where our understanding is so hazy that quantitative models would be premature.
All this will take painstaking structural changes to the way we teach and apply quantitative skills. Until then, the bulk of quantitative modeling will continue to lead to research that explains away systemic injustice and decision making systems that perpetuate it.
For example, datasets that are snapshots of a single system with many subjects at a single point in time tend to hide systemic discrimination whereas datasets that follow people over time as they are successively judged by many systems tend to reveal systemic discrimination.
Yet almost all available datasets belong to the former category as they are much cheaper to collect and serve the needs of the decision makers who fund this work. This is another way in which politics and ideology are encoded into seemingly inert and objective activities.

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Arvind Narayanan

Arvind Narayanan Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @random_walker

9 Aug
To better understand the ethics of machine learning datasets, we picked three controversial face recognition / person recognition datasets—DukeMTMC, MS-Celeb-1M, and Labeled Faces in the Wild—and analyzed ~1,000 papers that cite them.

Thread ⬇️
First, congrats to lead author Kenny Peng who worked on it for over a year. Kenny was a sophomore here at Princeton when he began this project, and he did this in addition to his coursework and several other research projects. The other authors are @aruneshmathur and me.
Finding 1: despite retraction, ​​DukeMTMC and MS-Celeb-1M are available through copies, derivatives, or pre-trained models. They are still widely used in papers. The creators (especially MS) simply took down the websites instead of making the ethical reasons for retraction clear.
Read 13 tweets
4 Aug
Interested in the impact of recommender systems on society? @elucherini4, @MatthewDSun, @aawinecoff, and I have software, papers, and a talk to share:
– T-RECS, a simulation tool for studying these questions…
– Accompanying paper… 🧵
– A short piece on methodological concerns in simulation research…
– A talk (by me) offering a critical take on research on filter bubbles…

Here's a blog post that provides an overview and context:…
The key rationale for this work is that phenomena such as algorithmic amplification of misinformation, filter bubbles, or content diversity in recommendations are difficult to study because they arise through repeated interactions between users, items, and the system over time.
Read 4 tweets
2 Aug
Can machine learning outperform baseline logistic regression for predicting complex social phenomena? Many prominent papers have claimed highly accurate civil war prediction. In a systematic review, @sayashk and I find these claims invalid due to errors.
We are not political scientists and the main point of our paper is not about civil war. Rather, we want to sound the alarm about an oncoming wave of reproducibility crises and overoptimism across many scientific fields adopting machine learning methods. We have an ongoing list:
Incidentally, we learned about one of the systematic surveys in the above list because it found pitfalls in a paper coauthored by me. Yup, even researchers whose schtick is skepticism of AI/ML are prone to overoptimism when they use ML methods. Such is the allure of AI.
Read 8 tweets
19 Jul
In my dream version of the scientific enterprise, everyone who works on X would be required to spend some percentage of their time learning and contributing to the philosophy of X. There is too much focus on the "how" and too little focus on the "why" and the "what are we even".
Junior scholars entering a field naturally tend to ask critical questions as they aren't yet inculcated into the field's dogmas. But the academic treadmill leaves them little time to voice concerns & their lack of status means that even when they do, they aren't taken seriously.
One possible intervention is for journals and conferences to devote some fraction of their pages / slots to self-critical inquiry, and for dissertation committees to make clear that they will value this type of scholarship just as much as "normal" science.
Read 4 tweets
1 Jul
We shouldn't shrug off dark patterns as simply sleazy sales online, or unethical nudges, or business-as-usual growth hacking. Dark patterns are distinct and powerful because they combine all three in an effort to extract your money, attention, and data.… Image
That's from a 2020 paper by @aruneshmathur, @ineffablicious, Mihir Kshirsagar, and me.

PDF version:…
At first growth hacking was about… growth, which was merely annoying for the rest of us. But once a platform has a few billion users it must "monetize those eyeballs". So growth hackers turned to dark patterns, weaponizing nudge research and A/B testing.… Image
Read 5 tweets
30 Jun
I study the risks of digital tech, especially privacy. So people are surprised to hear that I’m optimistic about tech’s long term societal impact. But without optimism and the belief that you can create change with research & advocacy, you burn out too soon in this line of work.
9 years ago I was on the academic job market. The majority of professors I met asked why I chose to work on privacy since—as we all know—privacy is dead because of the Internet and it's pointless to fight it. (Computer scientists tend to be technological determinists, who knew?!)
At fist I didn't expect that "why does your research field exist?" would be serious, recurring question. Gradually I came up with a pitch that at least got interviewers to briefly suspend privacy skepticism and hear about my research. (That pitch is a story for another day.)
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!