Thread by @jmspool

It’s 2019 and I just heard a speaker tell a design audience that “there’s research that shows you only need 5-8 users to find 85% of the problems in a design.”

I pine for the day when this myth finally dies.

Well, this blew up. Seriously, it stuns me we’re still talking about this in 2019.

So, let’s take apart what I’m hearing here.

First, there are lots of people who’ve heard misrepresented versions of the original research.

The original research where the idea first made it's appearance in 1993 as Nielsen and Landauer's seminal paper "A mathematical model of the finding of usability problems"

dl.acm.org/citation.cfm?i…

Now, that paper didn't say that you could find 85% of all problems with 5-8 users.

They had collected up a bunch of usability studies, looked at the discovery rate of problems during the study, and tried to find a curve that would predict when you'd get to the 85% point.

You can read the original paper here: peres.rihmlab.org/Classes/PSYC64…

Only 5 of the 11 studies studied in the paper were usability tests. The rest were heuristic evaluations (which are a completely different thing that needs to die, but that's a battle for another day).

Of the usability evaluations, the largest study had 30 participants. This was a very large study in 1993. (It's a very small study by today's standards. Except, probably by people who still believe the myth of 5-8 users is all you need.)

The 30 participants in that study found 14 usability problems. Less than 1/2 a problem per user.

The study with the most identified problems was 145. That had 15 participants. (The first user found 23 of the 145 problems.)

That study was of Microsoft Office. They selected 15 participants, watched each one for about an hour and found 145 problems.

Each participant did exactly the same tasks. They were recruited from the same pool of people.

They weren't asked to do what they normally do with the package. They may not have had any previous experience with the product.

In total, the 15 participants found 145 problems.

Microsoft office came out 3 years before, in 1989. In those 3 years, it was a popular product, approaching almost a million users. People used it for hours on end each day.

en.wikipedia.org/wiki/Microsoft…

Do we think there were only 145 problems in that product? No, of course not. There were thousands of problems.

But finding 145 problems with 15 participants in 15 hours of usability testing seems about right. (Assuming we can all agree on what a problem is. Which the CUE studies showed we can't, but that's a battle for another day.)

What Nielsen and Landauer said: if you plotted all these studies on a graph, you could get a mathematical model that would show when you'd stop seeing new problems watching them do the same tasks with participants from a narrowly defined selection pool.

It was 8 users with their data. Not with every test everywhere. But with the specific data in that study.

Later studies showed that the model didn't really hold up. (I even co-wrote one of those papers.)

In the subsequent years, the mathematical model of Nielsen and Landauer turned into the myth that you find 85% of all problem with only 5-8 users.

Nobody every said this. There's no science to prove this.

But everyone seemed to believe it.

That's the myth we're fighting today.

So, let's come back to last night. I was watching a presentation when a member of the UX community, who does respectable work, was giving a great presentation where I was nodding my head in complete agreement…

…until…

they said, "And there's proven research that shows that you only need 5 to 8 users to uncover 85% of the problems in your product."

I was like "Noooooo!"

I was very disappointed to hear this. Here we are in 2019 and someone is still telling audiences this.

This is a failure of our education. We let these myths propagate without having a way to explain this can't possibly be true.

Today, it's not unusual for products to have thousands of developers working on them for a year. Those products have thousands of features, functions, and options. They may have millions of users interacting with them every day.

There's no way 5 to 8 participants will uncover anything meaningful. You'll barely touch the surface. And the study is likely extremely biased, with a very homogeneous participant pool to draw from. (How many folks with varying disabilities are represented here?)

Yet, there's a constant desire for there to be "a number."

If eight isn't the number, what is, I'm being asked frequently.

There is no number. At best, you can continue to look for the point of least astonishment.

Years ago, Will Schroeder and I published a paper adopting the Nielsen/Landauer curve to provide a formula to predict how many users you'd need, based upon the cumulative problems you've found already.

After every session, put in the distribution of problems found so far.

And Voilà, the formula would tell you how many more studies you needed.

Unfortunately, it only worked if you had a completely representative participant population. If your participants didn't work all the different ways your users did, the formula would break.

That's because people are wide and varied. And the problem with the Law of Large Numbers: Extremely small percentages of large numbers are still large numbers.

If you have 10,000,000 users in the world and you fix a problem that only helps 80% of them, that's 2,000,000 users you're not solving for. The 80/20 rule hurts millions of people.

The 5-8 user myth doesn't help us with this at all.

We should also keep in mind that much of the thinking around the time Nielsen/Landauer was published had to do with costs of testing and costs of development.

Back in Ye Old'e Days, it was expensive to run usability tests. There weren't that many people who could do it, they required expensive facilities, and recruiting participants was very expensive.

We thought it was a luxury to get multiple or on-going studies.

Set up of a study was particularly expensive. One reason to optimize the number of participants was that the more participants you had, the cheaper the overall costs per participant.

But that didn't make sense if you wouldn't get value with the additional participants.

Development costs were expensive too. Changes could take a long time and involve a large budget. You needed to be sure you were making the right changes before starting.

None of this is true any more. User research is extremely inexpensive (except in rare cases, like studies on mid-ocean oil rigs). Software changes are cheap and testable prototypes are cheaper.

Optimizing the number of users per round of iteration is less of a worry.

It's quite feasible to test 1 participant in a single iteration, then make changes based on what you saw. If you got it wrong, no worries, you'll undo it in a future iteration.

Because development costs are low and startup costs for studies are virtually nil, single person iterations are quite feasible. This helped methods like RITE come into play, changing the landscape of how we do user research today.

en.wikipedia.org/wiki/RITE_Meth…

In today's environment, smart teams conduct continuous user research. Set up costs are practically eliminated because every research study flows from the last one. Development costs are tiny, because it's part of the ongoing agile backlog.

The conditions that made the 5-8 user myth so appealing are now gone.

This is why it's past time for the myth to die. It is only holding us back.