@random_walker's Threads
Four out of eight chapters of our book on fairness and machine learning are now available—halfway there! The newly released chapter is on testing systems for discrimination.. fairmlbook.org/testing.html
A few thoughts in this thread on why you might find this interesting and useful.
A few thoughts in this thread on why you might find this interesting and useful.
Algorithmic auditing has exploded in the last 5 years (there are about 50 studies in our references). This young community has a lot to learn from the classic audit studies that date back to the 70s. The entire first half of the chapter is on traditional tests for discrimination.
The algorithmic bias conversation needs to move beyond simple measures of disparity. We help the reader connect these measurements to moral notions of fairness, understand the harms that can result, and identify the sources and mechanisms that produce disparities.
Today I want to take a break from sharing research to share a personal story instead. It’s a story about my name, why I once decided to quit academia, why I came back, what I learnt from it, and why I’m grateful to have an audience here on Twitter.
Near the end of my PhD, I had three main publications. I ran into someone at a conference whom I knew from previous events. He asked me about my work, and I briefly mentioned my papers. His reaction was something I didn’t expect…
He’d seen and liked my papers, but the name hadn’t registered. He didn’t realize the 3 papers had the same first author and that it was someone he knew. He was embarrassed and apologetic. If he’d known, he said, he’d have advocated for me at his school for a faculty position(!)
Much of what’s being sold as "AI" today is snake oil. It does not and cannot work. In a talk at MIT yesterday, I described why this happening, how we can recognize flawed AI claims, and push back. Here are my annotated slides: cs.princeton.edu/~arvindn/talks… 
Key point #1: AI is an umbrella term for a set of loosely related technologies. *Some* of those technologies have made genuine, remarkable, and widely-publicized progress recently. But companies exploit public confusion by slapping the “AI” label on whatever they’re selling.
Key point #2: Many dubious applications of AI involve predicting social outcomes: who will succeed at a job, which kids will drop out, etc. We can’t predict the future — that should be common sense. But we seem to have decided to suspend common sense when “AI” is involved.
Of all the flawed arguments I hear from surveillance apologists, the worst one is this: in the tribal village, there was no privacy, only gossip. And now we’re creating the global village.
Well, there were also no antibiotics in the tribal village, just magic herbs.
Well, there were also no antibiotics in the tribal village, just magic herbs.
What do you even say to someone who insists on equating global surveillance capitalism with a prelapsarian village and doesn’t recognize that what (maybe) worked on a scale 150 people doesn’t work at a scale of 8 billion?
For one thing, tribal gossip was a way of keeping power in check, whereas centralized technological surveillance is an unparalleled tool of social control. There’s a reason we had to invent democracy along with various checks and balances as we scaled our societies up.
Important to name names. HireVue and Pymetrics are the top co's selling snake oil responsible for horror stories like these. Zero peer-reviewed evidence that this stuff predicts job performance. Millions of people screened out of jobs based on elaborate random number generators.
It's shocking how many companies are in this space and how much money they've raised. They are chasing clients aggressively. Job applicants being screened by bogus AI is about to get much, much more widespread. arxiv.org/pdf/1906.09208…
HireVue and pymetrics prominently advertise that their tests comply with the EEOC's "four-fifths" guideline for race/gender disparate impact. They then pitch it to clients as a way to *minimize legal liability*. Who cares if it makes any sense if it can help avoid lawsuits?!
When I joined Princeton, my colleagues pointed out that the university predates the government of the United States. Cool! What I didn't realize back then is that the university has had time to accumulate more layers of bureaucracy than the government of the United States 😬
There are so many committees that there is a Committee on Committees to decide who will be on which committee dof.princeton.edu/terms-and-defi…
I have wondered who decides who sits on the Committee on Committees but have not dared to ask because I suspect that whoever asks that question gets put on that committee.
This is the most interesting short piece I've ever read about gig work: nytimes.com/2018/08/18/opi…
The root cause of the rise of insecure and temporary work in the US is not technology, but rather a profound restructuring of jobs that began over 40 years ago.
The root cause of the rise of insecure and temporary work in the US is not technology, but rather a profound restructuring of jobs that began over 40 years ago.
Critics of the tech industry correctly like to call out the naive belief that technology will fix social problems. Yet those same critics (myself included 😬) are sometimes too eager to *blame* technology for social problems caused by bad decisions or bad policy.
I just found out that the author of the op-ed, @louishyman, has a book that fleshes out his thesis. Packed with insights. Purchased!
amazon.com/Temp-American-…
amazon.com/Temp-American-…
When companies like Equifax & Yahoo send out data breach settlement notices, they use new domains like equifaxbreachsettlement.com which ask users for PII to file a claim. It's impossible to tell if these are phishing sites.
New post:
freedom-to-tinker.com/2019/11/08/enh…
It gets worse… [thread]
New post:
freedom-to-tinker.com/2019/11/08/enh…
It gets worse… [thread]
In fact, Equifax's own customer service was directing people to a fake site for weeks: theverge.com/2017/9/20/1633…
The practice of using new domains also creates collateral damage by undermining decades of efforts to teach users to verify a site's identity before interacting with it.
The practice of using new domains also creates collateral damage by undermining decades of efforts to teach users to verify a site's identity before interacting with it.
To verify the ease of launching a phishing attack, we bought a fake domain for $15. We also sent a phishing email to a Yahoo mail account owned by ourselves and confirmed that Yahoo mail fails to flag it as spam.
Discussion paper: s3.amazonaws.com/citpsite/wp-co…
Discussion paper: s3.amazonaws.com/citpsite/wp-co…
I’m often told I’m "brave" for putting my papers on arXiv before they are accepted for publication. How silly! It’s 100x more likely for academic papers to languish in obscurity than for the ideas in them to be so good that someone would want to steal them.
When I ask those paranoid researchers if they’ve ever had an idea stolen, it always turns out to be something that happened to a friend of a friend of a friend, much like the urban legend of waking up in a bathtub full of ice with your kidney harvested.
The reason why ideas don’t get stolen is not that academia is a paragon of virtue. It’s more that someone plagiarizing from arXiv has a high risk of getting caught. Besides, ideas are a dime a dozen and everyone else is too obsessed with their own ideas to pay attention to yours.
This journalist went beyond "OMG they're fingerprinting us!"
They talked to the fingerprinting sites that they found, explained why it's bad, and got a few of them to promise to remove the fingerprinting 3rd parties. That's progress! More of this please. washingtonpost.com/technology/201…
They talked to the fingerprinting sites that they found, explained why it's bad, and got a few of them to promise to remove the fingerprinting 3rd parties. That's progress! More of this please. washingtonpost.com/technology/201…
Hey @geoffreyfowler thanks for your work and I hope you'll do a follow-up in a month to check if the sites that made promises followed through on them. And also ask the other sites if they've changed their minds after being named and shamed. 🤔
@geoffreyfowler Irony: if you clicked on the link, the Washington Post may have fingerprinted you. I can't tell if it's better or worse to fingerprint only users who've blocked cookies.
In 2013 I argued that Facebook, unlike previous social networks, had enough market penetration that it could never be outcompeted by another product simply by being better. The network effects are just too strong. Sadly, that's exactly how it's played out. randomwalker.info/publications/u…
In a previous 2012 paper, we analyzed dozens of decentralized social networks and concluded that:
1. they face fundamental barriers to adoption
2. contrary to the usual assumption, even if adopted widely, they wouldn't necessarily improve privacy.
randomwalker.info/publications/c…
1. they face fundamental barriers to adoption
2. contrary to the usual assumption, even if adopted widely, they wouldn't necessarily improve privacy.
randomwalker.info/publications/c…
Here's an excellent essay that helped me think about how people choose among social networks: "White Flight in Networked Publics?" by @zephoria. A purely economic analysis of social media platforms misses important dimensions of race and class. danah.org/papers/2009/Wh…
Contrary to endless "privacy is dead" predictions, people are making more privacy-conscious decisions in the last few years. Many companies now pitch privacy as a selling point: large ones like Apple, medium-sized ones like Firefox, and smaller ones like DuckDuckGo and Brave.
A nuance worth mentioning: DuckDuckGo and Brave were privacy-focused since their inception (DDG in 2008 and Brave in 2015) whereas Apple and Firefox have moved in a privacy-focused direction in response to shifting attitudes among users leading to a market opportunity.
There have been other recent wins for privacy: most notably, the GDPR. But consumer privacy regulation can only really succeed if it creates incentives that reshape the market. So it's reassuring to see privacy-focused tech companies flourish.
GDPR requires active consent for tracking. Most websites have pre-selected boxes, which is not allowed. What % of users would actively accept tracking to get the supposed benefits of personalization? 50%? 25%?
A new study found it's about ONE FIFTH OF 1%. arxiv.org/pdf/1909.02638…
A new study found it's about ONE FIFTH OF 1%. arxiv.org/pdf/1909.02638…
The tracking industry is acutely aware that the affirmative consent requirement is an existential threat. That's why cookie consent dialogs are loaded with dark patterns. The study confirms that even simple design changes are highly effective. arxiv.org/pdf/1909.02638…
Of course, why stop at basic tricks like pre-selecting boxes when you can get creative?
The platform company playbook: ignore abuses of the platform; leave it to customers/users, journalists, and 3rd parties to police harmful activities; wait until there's an outcry take stuff down. In other words, externalize the costs and keep the profits. cnbc.com/2019/10/20/ama…
Amazon has defended against lawsuits by arguing that it isn't the seller. But this is disingenuous:
–Amazon advertises itself to customers as a 1-stop shop
–It makes its own brand way more prominent than the seller's
–Amazon doesn't even know the seller's identity in many cases!
–Amazon advertises itself to customers as a 1-stop shop
–It makes its own brand way more prominent than the seller's
–Amazon doesn't even know the seller's identity in many cases!
Does the practice of burdening users and journalists with the responsibility of policing platforms sound familiar? It should, because many other companies operate that way.
This turned out to be relatable to people who have to use apps like Concur, but especially Electronic Health Record software. Doctors, I feel your pain! EHRs, I’ve learned, make all other enterprise software look like Microsoft Paint. The reasons are interesting and depressing.
BTW, here’s why I know anything at all about this: medical groups get pitches from blockchain companies claiming to fix EHRs 🙄 They sometimes call me—I teach blockchain tech—to ask if this makes any sense. I’ve used that as a chance to learn why EHRs suck in the first place.
To start, everything I said about Blackboard applies to EHRs: the people doing procurement are far removed from medical practice and evaluate them based on checklists. But there are at least three other factors that push EHRs into their own circle of hell.
My university just announced that it’s dumping Blackboard, and there was much rejoicing. Why is Blackboard universally reviled? There’s a standard story of why "enterprise software" sucks. If you’ll bear with me, I think this is best appreciated by talking about… baby clothes!
There are two types of baby outfits. The first is targeted at people buying gifts. It's irresistible on the rack. It has no fewer than 18 buttons. At least 3 people are needed to get a screaming baby into it. It's worn once, so you can send a photo to the gifter, then discarded.
Other baby outfits are meant for parents. They’re marked "Easy On, Easy Off" or some such, and they really mean it. Zippers aren't easy enough so they fasten using MAGNETS. A busy parent (i.e. a parent) can change an outfit in 5 seconds, one handed, before rushing to work.
Vocabulary choices shape public debates. Imagine if the vague term "AI" didn’t exist and machine learning were known by the more honest term applied function approximation. That would expose the absurdity of "strong AI" predictions—it’s as likely as building a ladder to the moon.
My point isn't that strong AI is impossible. My point is that while the current spurt of progress in ML is really cool, we have absolutely no reason to think that it constitutes meaningful progress towards Artificial General Intelligence.
Ha, it looks like others have made the same analogy:
A paper by Beijing researchers presents a new machine learning technique whose main uses seem to be trolling and disinformation. It's been accepted for publication at EMLNP, one of the top 3 venues for Natural Language Processing research. Cool Cool Cool
arxiv.org/pdf/1909.11974…
arxiv.org/pdf/1909.11974…
The paper has no discussion of potential unintended uses (actually it doesn't say what the *intended* uses are except, tautologically, news comment generation). Unfortunately, EMNLP does not seem to require authors to address or acknowledge the ethical implications of their work.
Computer security has a long history of double-edged consequences of research. Security conferences these days require submissions to describe ethical considerations and how the authors followed ethical principles. Machine learning conferences should consider doing this.
When we watch TV, our TVs watch us back and track our habits. This practice has exploded recently since it hasn’t faced much public scrutiny. But in the last few days, not one but *three* papers have dropped that uncover the extent of tracking on TVs. Let me tell you about them.
The first paper looked at Roku and Amazon Fire TV. These platforms let you subscribe to “channels”, which are basically apps. As you can guess, they are loaded with trackers. Doubleclick alone is on 97.5% of Roku channels. tv-watches-you.princeton.edu/tv-tracking-ac…
There are some channels with over 50 trackers. Also, the majority of trackers were able to grab a unique ID such as MAC address. A few channels leaked email addresses to trackers and many leaked video titles—often unencrypted, so your viewing history is exposed on the network.
I'm starting to suspect that people who tend to use metaphors like three dimensional chess, four dimensional chess, or especially twelve dimensional chess (!) have never actually tried playing a game of plain old two dimensional chess.
Chess isn't tic tac toe, people! It's almost certainly more complex than whatever mundane thing is being described as fourteen dimensional chess.
By the way, even three dimensional tic tac toe is too hard for humans en.wikipedia.org/wiki/3D_tic-ta…
Whenever someone points out sexist/racist stereotypes in a new AI tool, you’ll find lots of apologists in the comments saying, "What's the surprise? It just reflects the training data".
Well, the "surprise" is that researchers keep releasing these tools as if everything’s fine.
Well, the "surprise" is that researchers keep releasing these tools as if everything’s fine.
To OpenAI's credit, they released a (rudimentary) model card based on the "Model Cards for Model Reporting" paper github.com/openai/gpt-2/b…
arxiv.org/pdf/1803.09010…
Sadly, it's unlikely that most people who downloaded the model even noticed the model card, let alone abided by it.
arxiv.org/pdf/1803.09010…
Sadly, it's unlikely that most people who downloaded the model even noticed the model card, let alone abided by it.
OpenAI said they wouldn't release one of their models because it could be used by political actors to generate seemingly realistic disinformation text. This follows a long line of claims that AI is dangerous because it's too good, when in fact it's harmful because it's too crude.
Many of today's prominent computer scientists entered the field decades ago, inspired by the potential of digital technology to hold powerful institutions to account. But gradually tech has become an instrument of power. The tech industry itself badly needs more accountability.
To stay true to our ideals, computer scientists—especially in academia—must reexamine our cozy relationship with the industry. We must broaden our focus: in addition to building technology, we need to incorporate critical thinking about its double-edged effects on society.
The CS community has responded to ethical concerns by promoting a distinct field, algorithmic fairness. There are many questionable things about this approach. "Fairness" is a narrow conception of ethical responsibility, and it tends to get further reduced to a technical problem.
Universities seem to have many rules carefully designed to worsen students’ educational experience and outcomes. Here are some examples. Please share more!
(Yes, there are bureaucratic reasons for these rules. But what matters is the actual effect, which end up being perverse.)
(Yes, there are bureaucratic reasons for these rules. But what matters is the actual effect, which end up being perverse.)
PhD scholars must do their internships in the summer, even if they’ve completed coursework and the academic calendar is no longer relevant. Summer is usually the worst time for an internship because the advisor is (finally!) available for one-on-one mentoring and collaboration.
Classrooms are required to be larger than the enrollment limit. But lots of students unenroll after shopping for classes, so actual attendance is far lower. The result is large, empty lecture halls, guaranteed to put students to sleep.
When a company that has your personal data reneges on promises about what it will/won't do with the data, it should be required to prominently disclose this by *rebranding*, not just adding words to a privacy policy. minnesotalawreview.org/wp-content/upl…
A provocative 2013 paper by @paulohm.
A provocative 2013 paper by @paulohm.
@paulohm Perhaps we can start with something far less ambitious: the branding of apps and products should disclose which company owns them. If a seemingly hip privacy-friendly product is made, owned, or bought by a creepy surveillance capitalist, we want to know.
@paulohm Facebook announced it is adding the Facebook brand to WhatsApp & Instagram — "WhatsApp from Facebook" / "Instagram from Facebook". In principle, this is good. It will make it harder to isolate those brands from the impact of FB's privacy scandals. Except… can you even spot it?
