Thread by @doctorow

In fall 2020, Facebook went to war against Ad Observatory, a NYU crowdsourcing project where users capture paid political ads through a browser plugin that santizes them of personal info and uploads them so disinformation researchers can analyze them.

pluralistic.net/2020/10/25/mus… 1/

Facebook's attacks were truly shameless. They told easily disproved lies (for example, claiming that the plugin gathered sensitive personal data, despite publicly available, audited source-code that proved this was absolute bullshit). 2/

Why was Facebook so desperate to prevent a watchdog from auditing its political ads? Well, the company had promised to curb the rampant paid political disinformation on its platform as part of a settlement with regulators. 3/

Facebook said that its own disinfo research portal showed it was holding up its end of the bargain, and the company hated that Ad Observatory showed that this portal was a bad joke:

pluralistic.net/2021/08/06/get… 4/

Facebook's leadership are accustomed to commanding a machine powerful enough to construct reality itself. 5/

That's why they nuked Crowdtangle, their own internal research platform that disproved the company's claims about how its amplification system worked, showing that it was rigged to goose far-right conspiratorialism:

pluralistic.net/2021/07/15/thr… 6/

And while Facebook claims that it wants to purge its platform of disinformation, the reality is that disinfo is *very* profitable for the company. 7/

Ads for financial fraud, identity theft, dangerous scam products, and political disinformation are disproportionately lucrative for Facebook:

pluralistic.net/2020/12/11/num… 8/

All of this is the absolutely predictable consequence of Facebook's deliberate choice to "blitzscale" to the point where they are moderating three billion users' speech in more than 1,000 languages and more than 100 countries. 9/

Facebook may secretly like failing at this, but even if they were serious about the project, they would still fail. 10/

Whenever Zuck is dragged in front of Congress and they demand answers about what he's going to do about the open sewer he's trapped billions of internet users in, he always has the same answer: "The AI will fix it."

washingtonpost.com/news/the-switc… 11/

This is the pie-in-the-sky answer for every billionaire grifter (see also: "How will Uber ever turn a profit?"). No one who understands machine learning (except for people extracting fat Big Tech salaries) takes this nonsense seriously. They know ML isn't up to the job. 12/

But even by the standards of machine learning horror stories, the latest Facebook moderation failure is a fucking doozy. Genocidal, even. 13/

Remember when Facebook management sat idly by as its own staff and external experts warned them that the platform was being used to organize genocidal pogroms in Myanmar against the Rohingya people? Remember Facebook's teary apology and promise to do better? 14/

They didn't do better.

The human rights org @Global_Witness tried buying ads on Facebook for eight pro-genocide phrases that had been used during the 2017 genocide. 15/

Facebook accepted *all eight ads*, even though they duplicated the messages it promised it would block in the future (Global Witness cancelled the ads before they could run).

apnews.com/article/techno… 16/

Some of the phrases Facebook's moderation tool failed to catch:

* "The current killing of the [slur] is not enough, we need to kill more!"

* "They are very dirty. The Bengali/Rohingya women have a very low standard of living and poor hygiene. They are not attractive" 17/

Facebook has claimed that:

a) It will filter out messages that promote genocide against Rohingya people;

b) It will subject paid ads to higher levels of scrutiny than other content;

c) It will subject political ads to the highest level of scrutiny. 18/

Facebook used legal threats to terrorize accountability groups seeking to hold them to these promises, stating that its in-house tools were sufficient to address its epidemic of paid political disinformation. 19/

A common newbie error in machine learning is to forget to hold back training data to evaluate the model with. 20/

Training an ML model involves feeding it a bunch of data (say, "messages that foment genocide against Rohingya people") so it can build a statistical model of what its target looks like. 21/

Then you take some of that training data - a portion you didn't use to train the model on - and see if the model recognizes it. 22/

If you forget and evaluate your model using some of its training data, you're not measuring whether the model can evaluate *new* input correctly - you're just checking to see whether it remembers seeing this input it's already seen. 23/

Incredibly, FB seems to have done the opposite. 24/

They made a filter than *can't recognize the input it was trained on.* It didn't need to make any inferences about whether "we need to kill more" was a genocidal message, because it had been shown a copy of that message bearing the hand-coded label "genocide." 25/

This is the kind of fuckup you have to work hard to achieve. It's galaxy-class incompetence. And it's about genocide, in a country currently under martial law, where Facebook already abetted one genocide. 26/

Even by the low standards of FB, this is a marvel, an 85,000 Watt searchlight picking out the company's dangerous incapacity to take rudimentary measures to prevent the kinds of crimes against humanity that are the absolutely foreseeable consequences of its business model. 27/

Image:

Anthony Quintano (modified)
commons.wikimedia.org/wiki/File:Mark…

CC BY 2.0:
creativecommons.org/licenses/by/2.…

Japanexperterna.se (modified)
flickr.com/photos/japanex…

CC BY-SA 2.0:
creativecommons.org/licenses/by-sa…

Cryteria (modified)
commons.wikimedia.org/wiki/File:HAL9…

CC BY 3.0:
creativecommons.org/licenses/by/3.… 28/

ETA - If you'd like an unrolled version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

pluralistic.net/2022/03/23/fal…