Thread by @Pinboard

I attended one of these meetings (working on an article that later got spiked), listened to FBI, big tech, and privacy advocates all speak up, and was very impressed with how it was conducted. The issue is genuinely difficult. Alex Stamos's thread here is very much worth reading.

https://twitter.com/alexstamos/status/1424054547362697216

The ability to share images and video worldwide is unfortunately also a driver for child abuse. Every large site that lets people upload photos and video runs into this fact. The current arrangement (involving NCMEC, fingerprinting, and big tech companies) is a fragile tradeoff

The governance problem here is we have six or seven giant companies that can make unilateral decisions with enormous social impact, and no way of influencing those decisions beyond asking nicely for them to come talk to the affected parties before they act.

Since these platforms and devices are global, sometimes that impact takes place in contexts that none of the people working on the technology even know exists. If Facebook moves groups to full E2E, or Apple shifts content monitoring from to devices, the decision is unilateral

The way we find out about these technology impacts is by rolling them out worldwide and then seeing what social and political changes result. It's certainly a bracing way to run experiments, with no Institutional Review Board to bog everything down with pessimism and bureaucracy

But the problem is there's no way to close the loop right now, to make it so that if Apple or Facebook or Google inflict huge social harm, their bottom line suffers, or their execs go to jail, or they lose all their customers. Profits accrue while social impacts are externalized

One way out is to keep companies small enough so that their decisions don't affect billions of people. Another is to start breaking links instead of connecting everything. However we do it, the era of being able to YOLO global public policy in Cupertino or Palo Alto needs to end.

With the Apple thing, there have been at least three discussions smashed into one:

1. Who's in charge of declaring content illegal/who does the phone snitch to?
2. What's the right tradeoff for fighting CSEM?
3. What vulnerabilities will this architecture introduce to phones?

And of course we're left discussing these difficult issues on global platforms that were themselves rolled out as social experiments, and have since optimized themselves to reward performative strife. So it all gets a little hard to handle.

The problem here is a subtler version of the cryptocurrency problem—being able to write code should not give you a pass from having to participate in the messy process of social consensus, put you beyond the law, or exempt you from democratic checks on concentrated power.

Facebook's reaction to the governance problem has been to cosplay a Supreme Court. Apple's and Google's reaction has been a raised middle finger. So far it's only China that has successfully made Apple bend the knee and adapt their designs to fit the CCP's desired social outcomes