,
5 tweets,
1 min read
Read on Twitter
Some news: I’m writing a book for @nostarch titled “The Machine Learning Red Team Manual”. My aim is to provide a practical guide for anyone interested in adversarial ML and red teaming as it relates to in-production ML systems. A short thread on why this project matters:
ML is the hot thing to integrate into products and services across many different sectors of the economy. However, systems predicated on ML have unique security considerations: vulnerabilities are present at both the algorithmic and systems levels.
Many media outlets have picked up on algorithmic-level vulnerabilities through academic research on adversarial examples. These papers are fascinating because they tell us how machine intelligence differs from human intelligence but they tell us nothing about security.
Deployment context matters a lot when thinking about security. Many papers (not just adv example papers) suggest cool algorithmic attacks on ML but focus on constrained problems that don’t transfer to real systems due to unrealistic assumptions about attacker-controlled assets.
Threat modeling and risk analysis are key to security but are absent in academic adversarial ML literature because the focus there is to explore, not to provide security recommendations. My book aims to bridge this gap between academic research and secure system deployment.
