Thread by @tdhopper

Nine Challenges in Applying Machine Learning to Cybersecurity

1️⃣Defining the problem
2️⃣Labeled data
3️⃣Cost of being wrong
4️⃣Model decay
5️⃣Adversarial actors
6️⃣Biased feedback Loops
7️⃣Data access
8️⃣Model interpretability
9️⃣Model instability

1️⃣ Defining the problem.

You can't accurately mitigate malicious activity unless you can define malicious activity.

2️⃣ Acquiring labeled data.

At some point, all machine learning methods need a sizable amount of labeled data. This relies on 1️⃣ and highly trained, human labelers.

3️⃣ Quantifying costs of bad predictions.

Most ML libraries default to assuming false negative and false negatives have the same cost. This is rarely true in security problems. Accurate quantification is difficult.

(PS online learning is really hard.)

4️⃣ Model drift.

Data generated from cyber-systems (especially those being influenced by adversarial actors) aren't stationary! Models that work now, won't work eventually.

5️⃣ Adversarial actors.

Bad actors have an incentive to bypass your models. Repeated access to model predictions allows these actors to learn about your models.

6️⃣ Feedback loops.

Mitigating malicious activity changes the nature of your data and makes it difficult to retrain. (Read this! ai.google/research/pubs/…)

7️⃣ Data access.

By the very nature of security, data (essential for training) will be difficult to access. (Even before GDPR.)

8️⃣ Model interpretability.

Customers, regulators, analysts, CEOs may demand explainability. Many models are hard to explain. This may restrict your space of features and models.

9️⃣ Non-Pareto Model Transitions.

Updating (retraining) machine learning models can introduce “regressions” in the classifications. How do you update your model without negative consequences (e.g. non-malicious things being labeled as malicious.)

my soundcloud tdhopper.com/talks