Everyone keeps writing these doomed takes about how “the US government is going to force tech companies to comply with surveillance, so they might as well just give in preemptively.” Like it’s inevitable and we should just hope for what scraps of privacy we can.
Even I was pessimistic last week. What I’ve seen in the past week has renewed my faith in my fellow countrymen — or at least made me realize how tired and fed up of invasive tech surveillance they really are.
People are really mad. They know that they used to be able to have private family photo albums and letters, and they could use computers without thinking about who else had their information. And they’re looking for someone to blame for the fact that this has changed.
People are telling me that Apple are “shocked” that they’re getting so much pushback from this proposal. They thought they could dump it last Friday and everyone would have accepted it by the end of the weekend.
I think that reflects Apple accepting the prevailing wisdom that everyone is just fine having tech companies scan their files, as long as it’s helping police. But that’s not the country we actually live in anymore.
Anyway, I don’t revel in the fact that Apple stuck their heads up and got them run over by a lawn mower. I like a lot of the people on Apple’s security team (I turned down a job there a few years ago.) But people need to update their priors.
At the end of the day, tech companies do care a lot about what their users want. Apple has heartburn about this *not* because Congress passed a law and they have to do it. They’re panicked because they did it to themselves, and they can’t blame Congress.
A few folks in Congress, for their part, have been trying for years to pass new laws that force providers to include mandatory backdoors like this new Apple one. They failed repeatedly. In part they failed because these systems aren’t popular.
And so the shell game has been to play one against the other. Congress can’t quite pass laws requiring backdoors because there’s no popular support. But providers somehow have to do it voluntarily because otherwise Congress will pass laws.
And as an addendum: tech companies are incredibly risk averse, stupid beasts. They don’t take risks. They don’t stick their necks out. They do the thing that seems safe, and collaborating with law enforcement always *feels* safe.
(Dumb tech companies are filled with brilliant people. But the basic economics tell them to be conservative and stupid.)
• • •
Missing some Tweet in this thread? You can try to
force a refresh
I’m not denying that there’s a CSAM problem in the sense that there is a certain small population of users who promote this terrible stuff, and that there is awful abuse that drives it. But when we say there’s a “problem”, we’re implying it’s getting rapidly worse.
The actually truth here is that we have no idea how bad the underlying problem is. What we have are increasingly powerful automated tools that detect the stuff. As those tools get better, they generate overwhelming numbers of reports.
Someone pointed out that Apple’s Intel Macs probably can’t run their client-side scanning software because they don’t possess a neural engine coprocessor. Real time scanning on Macs is going to require an upgrade to newer M1 hardware (or beyond).
It’s sure a weird thing to pay a ton of money for Apple’s latest hardware, and the first thing they do with it is scan your personal files.
Some other folks have asked whether corporate and enterprise-managed devices will be subject to scanning. What I’ve heard is that enterprise customers are *very* surprised and upset. Apple hasn’t announced if there will be an MDM setting to disable it.
It’s gradually dawning on me how badly Apple screwed up with this content scanning announcement.
If Apple had announced that they were scanning text messages sent through their systems, or photo libraries shared with outside users — well, I wouldn’t have been happy with that. But I think the public would have accepted it.
But they didn’t do that. They announced that they’re going to do real-time scanning of individuals’ *private photo libraries* on their own phones.
That’s… something different. And new. And uncomfortable.
Yesterday we were gradually headed towards a future where less and less of our information had to be under the control and review of anyone but ourselves. For the first time since the 1990s we were taking our privacy back. Today we’re on a different path.
I know the people who did this have good intentions. They think this was inevitable, that we can control it. That it’ll be used only for good, and if it isn’t used for good then that would have happened anyway.
I was alive in the 1990s. I remember we had things like computers that weren’t connected to the Internet, and photo albums that weren’t subject to continuous real-time scanning. Society seemed… stable?
Reading through the analysis. This is not… a security review.
“If we assume there is no adversarial behavior in the security system, then the system will almost never malfunction. Since confidentiality is only broken when this system malfunctions, the system is secure.”
Don’t worry though. There is absolutely no way you can learn which photos the system is scanning for. Why is this good? Doesn’t this mean the system can literally scan for anything with no accountability? Not addressed.
A small update from last night. I described Apple’s matching procedure as a perceptual hash function. Actually it’s a “neural matching function”. I don’t know if that means it will also find *new* content on your device or just known content.
Also, it will use a 2-party process where your phone interacts with Apple’s server (which has the unencrypted database) and will only trigger an alert to Apple if multiple photos match its reporting criteria.
I don’t know anything about Apple’s neural matching system so I’m hopeful it’s just designed to find known content and not new content!
But knowing this uses a neural net raises all kinds of concerns about adversarial ML, concerns that will need to be evaluated.