This is my team's work, and here is some more insight into how we got this done. What did we do beyond being Applied ML Engineers?
1. In 2014, we re-jiggered our strategy:
- No more anomaly detection. Only Security Interesting alerts
- Put domain knowledge front and center of our ML systems. Yes, LSTMs are cool, but the alerts better make a SecOps person happy.
blogs.msdn.microsoft.com/azuresecurity/…
- No more anomaly detection. Only Security Interesting alerts
- Put domain knowledge front and center of our ML systems. Yes, LSTMs are cool, but the alerts better make a SecOps person happy.
blogs.msdn.microsoft.com/azuresecurity/…
2. SecOps is not our turks for labeled data; they are our partners and solid source of domain knowledge. As Applied ML Engineers, we sit right next to them and triage alerts every single day; we go on hunts with them; we sit with them during incidents and be on call for them
3. Another source of domain knowledge, is service owners. Azure is complex, but those who built these systems sit around us. So, we would build our game plan for detections to protect Azure's crown jewels, with their input.
4. Another source of domain knowledge, is red team members. The amount of time I have walked to @sachafaust to brainstorm or @darkpawH to learn about pwning the cloud (and to get brownies) is well worth it. So, yeah, you wanna get these people lunch/beer regularly
5. Finally, lots and lots of testing. We test our detection before, during and post deployment using everything from previous pentest data to actual incident data to automated test harnesses. If something did not make the call volume bar, it is not being shipped.
6. As @SwiftOnSecurity noted problems rarely have simple solutions. Detecting geo login anomaly sounds really easy (I always logs in from Redmond, now logging in from Russia), but it could be because of travel, VPN, Company proxies and host of other things. Devil's in the detail!
