Thread by @doctorow

The Big Tech platforms can be horrible places. Harassers, abusers and griefers have figured out how to use them to meet one another, form vicious assault squads, and drive their targets off the service and make life miserable for those who stay.

themarysue.com/phd-in-gamerga…

1/

What's more, the platforms have so little competition - and are so siloed from one another - that leaving a platform comes with a heavy price, separating those who depart from their families, communities and customers.

eff.org/deeplinks/2021…

2/

With such high stakes and so many terrible actors, it's natural that the platforms all have account suspension and account termination policies so they can kick the worst offenders off their services.

3/

But these policies have an obvious weakness: they can be abused by harassers to trump up cases against their victims and get them terminated, or get their content removed.

4/

The platforms' natural solution to this has been to add safeguards to their policies, making them harder to invoke, creating ever-more-specific criteria and procedures for takedown, suspension and termination.

5/

Likewise, the platforms armored their put-back and account restoration policies, lest harassers figure out how to game them, returning to revictimize their targets. So takedown and put-back and termination and restoration have grown more complex and esoteric over time.

6/

This sounds like common sense, but it's a Red Queen's race, where you have to run faster and faster to stay in the same place. The thing is, harassers are dedicated to understanding these rules - that's their whole thing. Their victims just want to use the service.

7/

So harassers become rules-lawyers. They know exactly which phrases NOT to use to avoid a ban, and they know which phrases to invoke to get their accounts back. They have sprawling forums dedicated to developing and refining tactics to game platforms' policies.

8/

If you know where the tripwires are, you can avoid them - but you can also use them to trip your opponents. You can tiptoe up to the line and goad your victims into crossing it, then nark them out, with the exact phrases that get them sanctioned.

pluralistic.net/2020/09/18/the…

9/

This kind of abuse is present in every mass-scale removal and termination system. Think of the blackmailers who figured out how to use Youtube's copyright termination policies to extort creators' wages from them with threats of copyright complaints.

techdirt.com/articles/20200…

10/

And the companies' response to each abuse scandal is to make the policies more complex, adding new procedures to paper over the holes in the old ones. Those new procedures have their own holes, and so more patches are applied.

11/

We all know that if you swallow a spider to catch a fly, you'll have to swallow a bird to catch the spider, and so on. You can't fix a complex system's defects by adding more complexity.

12/

Online services create a monoculture, where a single set of policies control all our outcomes. As we know from agriculture and forestry, monocultures attract rich, parasitic ecosystems of specialized attackers, each with its own niche.

13/

So it is with account termination. Termination systems aren't just abused by griefers - they're also abused by professionals who establish account termination as a service businesses that sell Instagram bans for as little as $7.

vice.com/en/article/k78…

14/

These services have a pretty straightforward methodology: they create an account that matches all the personally identifying information of the victim, then claim that the victim is an identity thief.

15/

Because these are professionals, it's literally their job to know how to present and defend an identity theft case - while the victims are just everyday Instagram users, mired in the complex systems created to fend off scammers.

16/

The ban-as-a-service market is specialized. There's bottom-feeders who'll do the job for 6 euros; others sell to blackmailers who extort social media influencers by taking down accounts with millions of followers, with a sliding scale based on follower count.

17/

Given that sophistication, it's not surprising that @josephfcox was able to link some of these ban-vendors to services that charge thousands of dollars to expedite ban-reversals, capitalizing on their esoteric mastery of platform policies to get your account restored.

18/

The fact that account restoration costs hundreds of times more than banning teaches us a few things about the political economy of platform warfare, starting with the Kafkaesque nightmare that is account restoration.

19/

It's so bad that Facebook users who lost their accounts started buying $300 Oculus VR headsets in the (seemingly mistaken) belief that this would get them human attention, bypassing the unnavigable automated appeal system.

npr.org/2021/08/02/102…

20/

But the disparity also tells us that users value their accounts FAR more than attackers prize the ability to banish their targets. No wonder - the platforms didn't monopolize social media; they're into home automation, cloud services, online retail, payment processing, etc.

21/

Which means that losing your account could brick your thermostat, or cut you off from your creative wages, shut down your business's website, or erase decades of family photos and correspondence.

eff.org/deeplinks/2021…

22/

There are ways this could be better - the platforms could have a duty to return your data to you if they terminate your access and they could be required to separate social account termination from across-the-board termination on all their services.

23/

But when we're talking about proprietary silos with hundreds of millions or billions of users, there's only so much room for improvement. As @mmasnick's Impossibility Theorem has it, "Content moderation at scale is impossible to do well."

techdirt.com/articles/20191…

24/

In other words, when it comes to the fairness that arises from a nuanced, situation-specific judgment - the only way to separate trolls from victims - scale is a bug and not a feature.

25/

It's a fool's errand to try to scale moderation and termination up to serve as full-fledged civil justice systems for a wholly owned corporate "country" that is more populous than any nation in world history, with hundreds of languages and millions of community norms.

26/

Far more plausible is scaling these giants DOWN to the point where it's at least possible to parse through conflicting claims about nuance and meaning - and also where a bad call doesn't cost the loser access to a full digital life.

27/

Maybe we could do that with federation, through interoperability mandates like the ones in the #ACCESSAct:

digital-lab.consumerreports.org/2021/06/15/ins…

28/

And maybe we could do it through merger reviews and even unwinding the mergers that got us to this situation:

whitehouse.gov/briefing-room/…

29/

But we have to stop adding complexity to systems in order to cure the problems of complexity! The more specialized knowledge to need to keep your account online, the more we'll all have to fear from griefers, harassers and trolls.

eof/

ETA - If you'd like an unrolled version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

pluralistic.net/2021/08/06/get…