Thread by @RohanIrvine

Final talk for Day 1 of Design Research is one of my favourite presenters @bjkraal

Ben Kraal, Director of Experience Design at Symplicit
—
Maths for Design Researchers

#dr2021

Ben's going to talk to us about numbers and measures and why stakeholders want design researchers insights to be measureable.

Ben often finds himself in meetings where stakeholders do not particularly find qualitative insights, and from the back of the room you hear "Yes, but how big was your sample size?" or "Was it statistically significant?"

When we come to present research to people it's not part of their daily work. They don't encounter the practice and outcomes like we do, and bring their own understanding to make sense of research.

It's likely they've had access to market research — maybe like a preferred PM poll.

They're not trying to make decisions based on this information, so in an organisation they need to make a decision after you've presented it.

A lot of designers might not be confident in presenting numbers to stakeholders, which then loses trust between stakeholders.

So what numbers do we need to present?

Why are particular number right or good and why do they help us make decisions?

Stakeholders like numbers because they have a trust problem. Institutions that are big enough for lots of people to work in them make it difficult to build enough trust individually and instead moved their trust to measurement.

In Pride & Prejudice — Wickam is a soldier, Darcy pays for a commission. Back in the day the only way to become and officer in the army was to purchase a commission to prove you were serious.

Soldiers were payed based on the spoils of war, so buying into the army was a way to measure 'skin in the game' for trust.

Since then we've gotten better at measuring stuff.

As we built bigger machines we got better at measuring more finely

Machines wear out, and we had to be able to measure more finely to know when the machine was wearing out faster than we could adjust it.

The variation in machine made goods, and to ascertain that variation became a significant advantage.

in 1908, as brewer in the Guiness factory published a mathematical technique to measure small samples of hops to make consistent beer at scale.

He created the T test which is still used today.

How to assess whether numbers are trustworthy:

Does this measure what it actually claims to do?

Lots of surveys are shady, some are built on nonsense.

If you look at academic papers you can find measures that are tested. But you have to be very statistically literate

measuringu.com is a fantastic resource for explaining this

For numbers to be good you need a benchmark.

A benchmark lets you answer a question "what does good mean in this context"

Clients and stakeholders LOVE benchmarks — it makes your number more independent and lets you tell a better story.

It delegates the trust decision to a time before you were there.

If you combine a benchmarked quantitative measurement with qualitative insights you can tell an comprehensive story.

Good numbers are easy to administer and easy to report.

The Single Ease Question

"How easy or difficult was this task?"

1 — 7

Very difficult to Very Easy

You have to use it EXACTLY as the literature says.

If you don't, it won't measure what it says it will.

You have to use it the way the practitioner does if you want to say it's valid.

The other end of the scale, the User Experience Questionnaire

26 questions.

You get a spreadsheet and you have to read the academic paper to make sure you're using it correctly.

You have to do more work to understand the outcomes but you can tell a much richer story with this one.

The System Usability Scale (SUS) dates back to the 80's

It's a 10 question survey

Here's the paper Ben is referencing: ncbi.nlm.nih.gov/pmc/articles/P…

It's an assessment of all the systems where SUS was used.

The dot is the median score, the bars are the confidence interval — the confidence interval was that 95% sure that if a new person were to take this the SUS their response would be within the two bars.

None of the systems actually passed the good line.

The power is in your hands.

If your stakeholders are engaged and smart, you will get better questions like "why is it like that" which you can answer with your qualitative research.

You might struggle to create numbers for some types of research.

Numbers need explaining, you can't just put a figure up on a slide and say "done". If you have a benchmark it becomes richer.

You might have to have your stakeholders advocate for your outcome, it's easier with good numbers.

Iterate and measure again works really well in agile development methods.

Your stakeholders ask for numbers because they have a trust problem, and some measurements rely on randomness being predictable.

When you have a quantitative response, use it to tell a story.

Thanks @bjkraal! Another enlightening presentation!

#dr2021

@threadreaderapp unroll please