Thread by @chrissanders88

Big news!

After a long wait, I'm excited to publicly release my doctoral dissertation, "The Analyst Mindset: A Cognitive Skills Assessment of Digital Forensic Analysts".

You can download it here: chrissanders.org/2021/12/disser….

In the accompanying blog post, I also talk a bit about how I came to this research area, why I think it's important, and a little bit of what's next. While my doctorate is a terminal degree, my dissertation is a beginning toward more things to come. 2/

With that in mind, let me walk you through a high level overview of my research and findings here. This will be a long thread and pretty high level since it's nearly a 200 page document. 3/

Keep in mind that this is a research document and not a teaching document. While I cover a lot of ground for this type of research, the findings are still narrowly scoped. They tell a story, but not a complete story. That's what my classes are for. 4/

In my literature review I describe the problem I'm trying to solve and why there is a gap in knowledge here. 5/

Cyber security is still young and very complex. The industry is in a state of cognitive crisis characterized, in part, by a lack of understanding skilled analyst performance and too much tacit knowledge. 6/

The effects of this cognitive crisis result in a skills shortage. Not a people shortage, a shortage of people with the right skills. We can see this in many places, but a common one is the lack of available entry-level jobs. 7/

I am mostly concerned with the investigative skillset -- the domain of the analysts who identify and understand attacks. I identify 3 roles that are primary investigation-focused and 3 roles that leverage some investigative skillsets in support of the broader investigation. 8/

Industry and academia stink at teaching people to be analysts. This is in part because the people who are good at investigative work are not great at explaining why they are good at it or how they do their jobs. I don't blame them... that's a hard thing to do. 9/

There is also no meaningful, detailed accounting of the cognitive skills that analysts rely on. Some govt led efforts have been made that relate (DoL CCM, NSA-CAE, NICE) but they aren't detailed enough or don't acknowledge the investigative skillset as a unique domain. 10/

Universities struggle here for many reasons. A lot of them treat security as a child of computer science and rely on those faculty to teach it. At best, security as a practice is a cousin of CS. It is a unique domain that warrants unique approaches. 11/

Universities also struggle to identify WHAT to teach. They may rely on industry surveys or the govt led efforts whose issues I've already highlighted in this threat. Some just assimilate strategies from other colleges (made up of the firs two things). 12/

And, of course, a lot of them don't have a program-wide strategy at all. They hire faculty and let them wing it based on their experience. That goes back to the first problem of analysts not being able to identify or explain those processes. 13/

At the industry level, a very small amount of work has been done to identify analyst cognitive skills but it usually lacks academic rigor, is heavily biased, and companies are unlikely to share that information. 14/

So, this is where we are. We need to be able to teach people how to be analysts but we don't know what to teach them. That is why my research was and is necessary. Let's talk about how I designed it. 15/

My research questions were:
1. What procedural skills do experienced digital forensic analysts use during investigations?
2. What decision-making skills do experienced digital forensic analysts use during investigations?
16/

I chose to focus on procedural and decision making skills because they are somewhat fundamental. I need to identify these to identify other more nuanced skills. Also, these combined can help produce a unified model. Decisions lead to procedures lead to more decisions, etc. 17/

Cognitive Task Analysis is a collection of techniques that allow researchers to uncover cognitive skills (things in your head) from practitioners. I used two techniques: PARI and CDM. 18/

PARI stands for Precursor, Action, Results, and Interpretation and it's designed to uncover procedural skills. For this, I designed a tabletop investigation scenario that participants worked through verbally with me. 19/

CDM stands for Critical Decision Method and its designed to uncover facets of decision making skills. For this, subjects described novel investigations they worked and I identified decision points and asked probing questions related to them. 20/

This diagram provides an overview of my research method. I conducted the PARI and CDM collection and analysis separately analyzing results within and across cases. Then I synthesized the results to build a model of skilled analyst performance. 21/

My experience as a practitioner first enabled this research method. Someone without that experience would not have been able to do things like designing a realistic tabletop exercise or speak at the level of detail required to understand the nuance of the analysts processes. 22/

Once I finished the analysis, I provided the results back to the analyst subjects for review and feedback. After all, they are experts. Member checking is super useful for qualitative research in situations like this and helps establish reliability. 23/

In total, I had 9 subjects:
3 triage/SOC analysts
3 incident responders
3 forensic examiners

I used this spread because I wanted to uncover investigative skills universal to these roles. This is the knowledge domain I'm establishing. 24/

On average, participants spend 86% of their day performing investigations. The average years of IT and Security experience were 10.8 and 13.2 respectively. The average age was 38.

Now for the part folks care about... results! 25/

Starting with procedural skills, I identified 308 investigative actions, 210 precursors, and 95 interpretations. From my decomposition of that data, I uncovered 4 unique skill domains: inquiry, evidentiary, anomaly detection, and network mapping / attack visualization skills. 26/

Inquiry skills were all about the analysts forming investigative questions based on evidence to decide where they want to go next. Questions manifested the results of analysts sensemaking of evidence and pointed them toward additional evidence. 27/

Said another way, analysts interpreted evidence, forecasted potential events, established a hypothesis, and decided which evidence could prove/disprove that hypothesis. The investigative question was the manifestation of that work. 28/

Investigative questions came in a few varieties. The most common were event-relative questions that were related to specific timeline events. They were usually preceding (leading to an event), succeeding (following an event), or about the context of the event itself. 29/

Analysts also asked proximate questions that combined preceding and succeeding questions (like, +/- 5 minutes from an event). This is how analysts typically establish correlating events between data sources. 30/

Another question type were capability matching questions, where analysts sought to confirm or refute the presence of a known threat capability. This meant trying to determine if specific malware, software, or threat actors were present on a system/network. 31/

Last were utility questions, whose purpose was to gather data needed to answer other questions. These demonstrate how analysts often leverage multiple evidence sources toward a single analysis goal. 32/

Along with these question types, I identified clusters of questions with related goals. I called these Directed Analysis Techniques (DATs). Based on the scenarios I provided, I identified 9 DATs. 33/

The DATs I identified were prevalence, directed execution, undirected execution, reputation, malware capability, lateral movement, staging and exfil, phishing, and account role analysis. These won't be exhaustive but I have a framework for identifying more now. 34/

Next were evidentiary skills. These were skills that analysts leveraged to interpret meaning from evidence to form and answer questions. Dimensions included interpretation, capability comprehension, collection, and manipulation. 35/

After that were Anomaly Detection skills that focused on analysts ability to perform pattern matching of many sorts to assess the disposition of events. I observed the use of many discrete techniques. 36/

Anomaly detection techniques included baseline comparisons, frequency assessment, excluding known benign, timing, and syntax. Again, not exhaustive (based on my scenario), but I have a framework for identifying more. 37/

Finally, there was some evidence of analysts using cognitive visualization skills to map networks and attackers' movement through them. This most closely resembled something akin to a graph database with nodes and edges but occurred at a few different levels of abstraction. 38/

Now, on to Decision Making skills! I identified 100 decision points across my subjects' scenarios. I focused on decision cues (things that led to decisions) and goals (what they accomplished). 39/

I uncovered four cue types. First were relational cues that indicated the presence of other, yet to be discovered relationships relevant to the attack timeline. 40/

Next were dispositional cues that indicated whether an event or relationship was suspicious or benign.

Third were novelty cues that indicated the presence of some unknown threat, capability, or technology that the analyst did not understand well. 41/

Fourth were operational cues that came from an external input and indicated a potential to affect the analyst's ability to conduct the investigation. 42/

All these cues were things analysts recognized that compelled them to make a decision (usually about what to do next). Part of the analyst's skill was their ability to recognize these cues for what they were. Notable here that the same data can serve as multiple cue types. 43/

Now for decision goals -- I found six goal types: identify relationships, assess threat capability, assess forensic capability, perform bulk collection, take response action, and seek advice or help. 44/

As you might imagine, certain types of cues often led to specific decision goals. For example, relational cues typically led to decisions whose goal was to identify a relationship. Similarly, novelty cues often led to decisions whose goal was to assess a threat capability. 45/

Now for the finale... 46/

By identifying procedural and decision-making skills I was able to create a model that explained the workflow of expert analysts. The model is called Diagnostic Inquiry. 47/

Diagnostic because the goal is to find out what happened and assess a disposition of those events. Inquiry because forming investigative questions is the mechanism that moves the process forward. 48/

This is a much more detailed version of the initial model of diagnostic inquiry that I have written about before, based on my findings in the dissertation research. 49/

Using this model, I am able to walk through the investigation process of analysts and effectively describe the skills they leverage to conduct their investigation. 50/

It's notable that when I teach the model, I use a simplified version that looks different from this one. That's part of the point of this research, to establish something for which to create mental models useful for teaching and learning. 51/

That's pretty much it from a high level (he says 50 tweets later). If any of this interests you, I encourage you to read the paper. My blog post contains a guide to relevant sections based on your role (analyst, educator, researcher). chrissanders.org/2021/12/disser… 52/

And finally, my research is self-funded. If you find value in any of it (and want to see more), consider registering for one of my online classes where this work manifests (networkdefense.co/courses/) or donating to @RuralTechFund. 52/52