Thread by @RobertMLee

The moment we’ve (@DragosInc at least) have been waiting for...the S4 ICS Threat Detection results. #S4x19

All the competitors in the space were invited. In the end, three stepped up. Kaspersky ICS, an open source tool team by an ICS sec analyst from an asset owner/operator, and Dragos. Because the others didn’t participate it turned from a competition to more of an evaluation #S4x19

“Claroty and Dragos stepped up early. We reached out to 20+ of the vendors and they all said no.” @digitalbond then notes that Claroty backed out a few weeks before the competition so it morphed to an evaluation. The challenge kicks off with Ron who put 500+ hours into making it

Ron notes that last year was 3Gb and smaller network and this year it was 300+ GBs and more real world. Additionally no baseline this year. Also added this year “rabbit holes” and real world threats that is “truly advanced threat scenario” #S4x19

(My note: it’s obvious the competition isn’t about who’s the best but evaluating vendor claims, I like that approach as many are questioning some of the claims that get made)

Ron calls out that Gravwell, FireEye, Schneider Electric, and Rockwell contributed data and tools and insights/time. Kudos to the folks trying to be helpful to the community

Notes the challenge is a bit harder than reality (my note: harder is better) five different zones from processing, refining extraction, and mining. Was from a real asset owner

The attack scenario revealed. Notes he put Havex and Stuxnet in the network just to add complexity (I.e. true positives but not actually the end result, which makes sense to his comment about harder than real world since it’s a multi incident multi threat actor case) #S4x19

First team results are the two members from the open source team. They used tools such as anti, SecurityOnion, SOF-ELK, tshark, and moloch. (My note: they approached it really well). The group saw what happened in the DMZ and some of the signatures. Didn’t get far but huge kudos

The fact that they stepped up - and got so much done was awesome. @digitalbond steps in “he’s a very talented person that did this and he noted how hard this was without commercial tools.” Good takeaway. You need people and tech. Again kudos to them.

Dale makes a great point. It’s not that they couldn’t have been successful. But at scale/speed they couldn’t have been effective and efficient. I’m impressed with how much they did though

Now up is Kaspersky ICS team. Playing the recorded video from using KICS. Videos are recorded and will be made public post conference. They found the Stuxnet with indicators and noted the vulnerability being used. Nice video, encourage watching after.

KICS team also saw the Havex malware. Methodology seems to be identifying the indicator in KICS and then pivoting to Wireshark for analysis (smart to not just jump straight into packets initially).

(My note: seems to be doing well at identifying indicators but not necessarily the other actions or piecing together the intrusion. Seems to be a fair tool to use for classic IDS functionality extended to ICS. Not a diss - that has value depending on your requirements).

KICS results shown (blue they found grey they didn’t). Ron notes they got some good finds including firmware events but had trouble linking things together. Again kudos to KICS team for stepping up.

(My note: not sure the picture represents the amount of work done. Short timeline, highly complex scenario, large data set, accurately identifying even enough to get started on an investigation is good).

Now up is the @DragosInc team @dan_gunter and Danny B. The video is playing now complete with narration walking through the incidents. Much more focus on piecing the incidents together and utilizing asset identification as well.

Dragos team highlights how it goes beyond indicators to identifying the TTPs of threats, I.e. threat analytics. A lot of context around the alerts. Uniquely the Dragos Platform offered up the analysts playbooks (step by step guidance on how to deal with the threat).

There’s also a workbench to work the case across analysts directly in the Dragos Platform. Video shows value of taking advantage of asset identification in investigations. Found the various incidents but also which ones were connected and how.

Dragos findings revealed. The team’s focus was obvious on linking an intrusion for an incident response focused. Found GreyEnergy, Havex, etc but picked one incident to follow through into the OT networks.

Ron notes the lack of false positives in comparison on how the data was approached in the Dragos Platform. Alert fatigue is a real challenge and without knowing there’s an incident it would be much harder to find one.

Kaspersky comes back on stage to have an open to discussion with Dale, Ron, and Dan Gunter from Dragos. Kaspersky analyst makes note that having a relationship with the customer and doing initial tuning is normally incredibly valuable; difficult on blind dataset.

Dragos analyst (Dan) notes that going in blind is pretty normal though in many environments and the tools need to help add context.

Dale comments accurately that there’s a lot of noise normally in environments; and that complaining about noise isn’t enough. Dan agreed and notes that’s why Dragos isn’t a huge fan of anomalies and where threat behaviors are more important and sees less noise

Dale asks about data challenges and aspects of analysis. I think I interpreted the question differently than the folks on stage. My comment is yes you do need to do analysis up front at the sites from the technology. Not human analysis but tech doing analysis - doesn’t require ML

I.e. analytics at the edge instead of pulling all data back

Ron notes that he tried to leave notes and screw with the competitors as much as possible. Funny - not exactly real world lol but funny.

I was bound to complain about something, so here’s my complaint: the challenge wasn’t “find the final step” it was “tell me what’s going on”. So not releasing a score but then saying who got closer to “the end” flavors this incorrectly. Each competitor excelled in key areas.

Kaspersky did a fantastic job about finding various events. Dragos did a fantastic job of piecing them together and assisting the investigation (map and playbooks). Your requirements will drive you down different approaches. And again kudos to all who stepped up. Fun game. #S4x19