Thread by @ZackKorman

I wrote this thread about phishing simulations. We’ve sent about 3m sims, so we have pretty good data. I want to show what “works”, what people fall for, who falls for sims, etc. Also, why do sims at all. Sorry it’s long and boring, but might be useful for a very niche audience.

DATA (1): To explain how our system works so you can understand the data: we figure out info about each user (role, software they use, locations, languages, etc) and use that to target sims. And based on how a person responds, that helps us decide the next sim to send.

DATA (2): I’ll write a separate thread sometime on the details of that scoring process. But as for the sims themselves, they’re a mix of emails and teams messages, and they try to get people to click, leak credentials, call numbers, reply, scan QR codes, etc

DATA (3): The sims also contain a wide range of “difficulty levels” based on different characteristic. Some are sent from [e-mail]. Others are sent from perfectly spoofed addresses. Some are one-offs, others are sequences (eg password reset followed by reminder)

DATA (4): All of this is to say that despite having a ton of data, the system is highly dynamic. We have to be careful doing analysis as if it’s static. It’s easy to see a trend that’s actually a property of the targeting. So, keep that in mind when evaluating what follows.

WHO (1): Who falls for phishing sims? This one is easy. Everyone. In the first six months, on average about 35% of users will fall for at least one sim. This stat removes the theory that phishing sims identify “weak links”. Everyone falls for something.

WHO (2): Not all departments are equal though. There’s no real surprises here. Sales sucks, obviously. HR is mediocre, and finance is consistency better than most. (Grain of salt on this, lots of factors at play here)

WHO (3): But also notice that IT is pretty middle of the pack. This raises the question of why? Surely IT is far more knowledgeable when it comes to identifying anyone else. So, what explains this? Why do people fall for phishing?

WHY (1): This, to me, proves that phishing isn’t about knowledge. I don’t think people fall for phishing because they don’t know how to spot phishing. It’s instead because they forget to pay attention. They get caught at the wrong moment and they act too quickly.

WHY (2): This is supported by another fact: The users that correctly answer a training question about subdomains still fall for phishing sims that use subdomains! So it can’t be a knowledge thing, because I know these people have the knowledge.

EFFECTIVENESS (1): So… why run phishing sims? What’s the point? It’s not to “teach” people because, as shown, it’s not a knowledge issue. And it’s not to find the “weak links” because everyone fails. So what are phishing sims even for?

EFFECTIVENESS (2): I don’t want to dive too deep here, but my approach has always been that it’s about reminding people to pay attention. I want people to fall for sims because it makes them go “oh yea, phishing is real”. That’s it.

EFFECTIVENESS (3): And before people send it to me: There was a study presented at Blackhat arguing phishing sims aren’t effective. But… the control group received phishing sims? What they really tested was the education part. So their finding isn’t about phishing sims at all.

WHAT WORKS (1): Moving on to what “works” in phishing, starting with sender. A fully spoofed sender (microsoft[.]com) has the same fail rate as something mlcr0soft[.]com, and even micro-soft-support[.]com. People only notice if it’s wild (scams4u[.]org).

WHAT WORKS (2): The same is true for sender name. It either is very obviously wrong or people don’t notice it.

WHAT WORKS (3): Design is a weird one. The more “wrong” an email looks, the less people fall for it. But the worst possible designs actually have a slight uptick. Some of that is people going “lol scam” (not a great idea). But also it’s that it gets attention.

WHAT WORKS (4): Grammar and typos follow a predictable pattern. The more errors, the less people fall for a sim. The same is true for how believable the scenario is (password resets are more believable than winning a billion dollars).

EXAMPLES (1): One of the highest fail rates comes from employee satisfaction surveys. People love giving feeeback about how much they hate their boss (and reading how much others hate their boss too). So they also love handing over login credentials if that’s what it takes.

EXAMLES (2): Government stuff that is just plain text is also brutal. It doesn’t matter that the government would never send you a link to click. People click like crazy, and go on to put in all of their most sensitive info.

EXAMPLES (3): For sales people, honestly everything works. They’ll fall for anything. Here’s an example of one that totally fooled someone on our own sales team last week. (This is a photo from our platform as I don’t have examples in my Teams messages, sorry)

REPORTING (1): Moving to reporting phishing emails. People actually love reporting phishing emails! It’s weird and I didn’t expect that at all. People think you have to train your staff to do this but that’s not true. They just do it.

REPORTING (2): Across all of our customers that use our custom report phishing button (which I loathe), reporting rates are pretty constant and pretty damn high. I strongly suspect people just click it on every email they dislike, phishing or not.

REPORTING (3): We can see this in our data. More people have reported phishing sims than have clicked, even though only a subset of customers even have this functionality enabled.

The end. Like I said, very boring but maybe useful to some people. Also, here’s a live feed of us sending sims and people interacting, because that’s at least marginally more fun (although less so on the weekend) pistachioapp.com/worldmap

Anyway, if there are any specific stats you’re wondering about let me know and I can check when I have time.