Thread by @chrissanders88

There's often interesting public discussion about vendor detection tools and what they detect vs expectations. There's some interesting decision making that happens behind the scenes at these vendors when it comes to how they manage detection signatures. A thread... 1/

At a vendor, when you build a detection ruleset for lots of customers, you have to approach things a bit uniquely because you don't control the network where these rules are deployed and can't tune them yourself. 2/

One facet of this challenge is a decision regarding how you prioritize rule efficacy...we're talking accuracy/precision and the number of false positives that analysts have to investigate and tune. 3/

There are two ends of the spectrum here. You can...

Write only very specific rules with no false positives. But, you'll miss things.

or

You can write rules with very wide coverage. But you'll get a lot of false positives that require tuning. 4/

Pretty early on, most vendors generally see this as a dichotomy, and a choice that they need to specifically make because the folks writing these rules need guiding principles to work from.

They usually choose the lower coverage / lower FP route. Here's why... 5/

First, they know realistically, that most customers can't tune the rules they provide. They simply don't have the staff, expertise, or even visibility. Yes, that probably indicates many other problems, even for true positives. 6/

Second, some customers refuse to tune the rules. They believe the vendor should do this for them. Of course, that's usually not realistic because this tuning requires detailed network knowledge and access to data the vendor doesn't have. 7/

Third, sales pressure. Execs make the call bc they don't want to lose sales from #1 and #2. They know that many customers will complain or kick the vendor out (whether justified or not). Less FPs > More coverage from their perspective. 8/

Of course, that line of thinking is usually optimizing for the short term. Opinions change when a customer gets popped bc a tool didn't have coverage for something. A lot of these folks only think quarter to quarter. 9/

Fourth, there is a precedent with this from AV vendors. While IDS kinda grew up looking down at AV, lots of modern companies have AV vendor roots and take a similar approach with their detection strategy. Likewise, many customers are not conditioned to expect differently. 10/

Fifth and finally, it's harder to build tools that give people what they need to tune the rules. We're talking exclusion lists, time and criteria-based suppressions, and so on. It's more UX and dev time. 11/

I led one of these rule teams for a large vendor, which is a pretty small club of people. I experienced a lot of these things and heard many similar stories from my contemporaries. It's usually some combination of them and not just one. 12/

I don't say this to take up for vendors -- because honestly, I think an approach focused exclusively on either strategy is problematic and holds the industry back. 13/

I think the best model involves multiple rule sets. One that is high efficacy and on by default, and a second with broader coverage that requires tuning. 14/

Of course, you have to give folks the tools to do that tuning. Again, that means having detection engineers and analysts in the room with UX people and allocating dev time to make it possible. 15/

BTW, you can often tell which approach vendors take through their marketing. If they market their low FP rates, it's probably because of less coverage. If they market catching everything, expect to do lots of tuning. 16/

Last but not least -- this is also where I remind folks that an alert is never an answer -- it's a question. It's the analyst's job to find the answers.

You can also do lots of things with detections beyond alerting. See here: chrissanders.org/2018/12/additi…. 17/

All that said, feel free to AMA about vendor level detection engineering processes and I'll do my best to answer as Twitter character limits allow :)