I pine for the day when this myth finally dies.
So, let’s take apart what I’m hearing here.
First, there are lots of people who’ve heard misrepresented versions of the original research.
dl.acm.org/citation.cfm?i…
They had collected up a bunch of usability studies, looked at the discovery rate of problems during the study, and tried to find a curve that would predict when you'd get to the 85% point.
Only 5 of the 11 studies studied in the paper were usability tests. The rest were heuristic evaluations (which are a completely different thing that needs to die, but that's a battle for another day).
The study with the most identified problems was 145. That had 15 participants. (The first user found 23 of the 145 problems.)
Each participant did exactly the same tasks. They were recruited from the same pool of people.
In total, the 15 participants found 145 problems.
en.wikipedia.org/wiki/Microsoft…
Later studies showed that the model didn't really hold up. (I even co-wrote one of those papers.)
Nobody every said this. There's no science to prove this.
But everyone seemed to believe it.
That's the myth we're fighting today.
…until…
I was like "Noooooo!"
I was very disappointed to hear this. Here we are in 2019 and someone is still telling audiences this.
If eight isn't the number, what is, I'm being asked frequently.
There is no number. At best, you can continue to look for the point of least astonishment.
After every session, put in the distribution of problems found so far.
Unfortunately, it only worked if you had a completely representative participant population. If your participants didn't work all the different ways your users did, the formula would break.
The 5-8 user myth doesn't help us with this at all.
We thought it was a luxury to get multiple or on-going studies.
But that didn't make sense if you wouldn't get value with the additional participants.
Optimizing the number of users per round of iteration is less of a worry.
en.wikipedia.org/wiki/RITE_Meth…
This is why it's past time for the myth to die. It is only holding us back.