Thread by @doctorow

We hear a lot about how algorithms "radicalize" people, a semi-mystical process whereby "Big Data" and "Machine Learning" turn people into radical zombies who do whatever the system bids. In reality, it's a lot more banal.

1/

Take @alreadyeli's journey into becoming a paid Amazon reviewer, posting inauthentic reviews to help vendors game the algorithm. He signed up for a site called Rebatekey, "which offers rebates between 5% and 100% in exchange for leaving a review."

onezero.medium.com/my-bizarre-sti…

2/

He contacted Rebatekey customer service over Facebook Messenger. Facebook then tagged him as someone willing to write reviews for money. Even sleazier companies than Rebatekey send ads to people in this FB category offering them money for paid reviews.

3/

That's it. That's radicalization. It's just targeting.

The median human buys one or fewer fridges in their life. Targeting fridge buyers is hard. The traditional method is stuff like billboards near airports.

4/

These ads represent a crude targeting method: "People who fly have money. Fridges cost money. Put ads for fridges near airports."

It's not very effective. Almost no one who sees the ad buys a fridge. It's like a 0.000000001% conversion rate.

5/

FB lets you target people who've looked up fridge reviews. They ALSO probably won't buy a fridge after seeing your ad, but their conversion rate will be more like 0.000001%, which is a thousandfold improvement over airport billboards.

6/

Companies would be nuts to turn up their nose at a thousandfold gain, even if the overall effect size is very small. So FB gets a lot of ad dollars.

This works for any hard-to-locate trait: want to find people who believe gender is a spectrum?

That's there.

7/

How about people who want to march through Charlottesville carrying tiki torches and cosplaying Civil War criminals? That's there too.

8/

Finding people with hard-to-find traits represents a profound change in our discourse, politics and commerce, for good and bad, but it's not a mind-control ray, it's just targeting. Big Tech isn't made of super-geniuses who can get the public to do your bidding.

9/

Big Tech is made up of ordinary mediocrities like you and me who will help you find otherwise difficult-to-address target audiences and show them messages that they find compelling.

10/

tldr:

* Big Tech doesn't make people receptive to your message

* Big Tech finds people who are receptive

11/

This whole process also has a security dimension: when we are exposed to a new message, we seek out external indications of its validity: we read reviews, we look at Wikipedia entries, we check ratings with neutral observers.

12/

These processes are imperfect, and what's more, their utility changes over time, illustrating one of the bedrock principles of #SecurityEconomics, a variation on Goodhart's Law: "Any measure becomes a target."

13/

My favorite example of this is Google's early history: the Pagerank alogrithm Larry Page invented made a key observation: that pages that had a lot of inbound links were likely to be highly relevant, because relevance was the only reason for one page to link to another.

14/

Pagerank TROUNCED every other search-ranking algorithm by slurping up all available web-pages and counting how many links went to each page on the web.

15/

But making links to webpages isn't hard. As soon as there was a reason to link to webpages other than relevance (getting highly ranked by Pagerank), people made linkfarms to link to their pages and climb the ranks.

16/

Over the years, Pagerank and "Search Engine Optimizers" have been in an arms-race in which Pagerank makes it more expensive to game its system, and SEOs discover cheap workarounds, and/or seek out higher-margin customers who don't mind paying to overcome the new barriers.

17/

Which brings me back to Eli Reiter, writing fake Amazon reviews. Amazon customers rely on reviews as an important quality signal. Highly ranked items generate more sales, but when these items are actually low-quality, they generate returns and reduce trust in Amazon overall.

18/

Amazon and cheap goods vendors have been locked in an arms-race forever. First it was authors creating fake accounts to say nice things about their own books. Then it was companies paying clickworkers to write reviews for products without buying them.

19/

This gave way to "brushing" - sending terrible merchandise to random people so Amazon would accept a review from someone else:

marthastewart.com/1531369/myster…

20/

Now scammers are paying people like Reiter to place orders for goods, accept delivery of them, and write reviews. The vendors who can afford this kind of expensive countermeasure must be commanding margins of sufficient size to make it worth their while.

21/

Big ticket items sometimes generate these margins, as to high-volume items - but one of the most reliable ways to improve margins is to skimp on quality. Worse goods, made in worse conditions, yield excess rents that can be mobilized to generate good reviews.

22/

That is, "every measurement becomes a target."

Image:
Rion, modified
en.wikipedia.org/wiki/File:Sock…

CC BY
creativecommons.org/licenses/by/3.…

eof/