Thread by @matthew_d_green

The EU’s “chat control” legislation is the most alarming proposal I’ve ever read. Taken in context, it is essentially a design for the most powerful text and image-based mass surveillance system the free world has ever seen.

This legislation, which is initially targeted at child abuse applications, creates the infrastructure to build in mandatory automated scanning tools that will search for *known* media, *unknown* media matching certain descriptions, and textual conversations.

The legislation is vague about how this will be accomplished, but the “impact assessment” it cites is not. The assessment makes clear that mandatory scanning of images & text, especially in encrypted data, is the only solution the Commission will consider.
eur-lex.europa.eu/legal-content/…

If you wonder what detecting “grooming behavior” means, here is a brief description. Roughly it means developing new AI tools that can understand the content of textual conversations and can automatically report you to the police based on them.

You might ask how the EU, famous for its focus on privacy, justifies the development of automated text-analysis tools that scan your private chats. The Impact Assessment has an analysis. To say that this analysis is deficient is really much too kind.

As a technologist I have to point out that the technological solutions to do this *safely* don’t exist. They are at best at the research stage. ML textual analysis schemes do exist, and often misfire. These systems will need to accomplish this task perfectly and also privately.

The idea that we can deploy AI systems to read your private conversations and report crimes is frankly dystopian. Even if such systems existed, no reasonable democracy would vote for this. But this is what the EU is proposing to mandate and *build* in the next couple of years.

If you take comfort from the fact that these systems are aimed at “awful crimes” or “will be fully transparent”, please don’t. The nature of these proposals is that they will be easy to reprogram, either by law or by technical accident.

For those who want to read for themselves, here is the very dry text: eur-lex.europa.eu/legal-content/…

And here is the impact assessment, which is much more descriptive: eur-lex.europa.eu/legal-content/…