Thread by @RohanIrvine

First talk of the day:

Vidhika Bansal, UX Group Manager at Intuit

— Pensieves & Crystal Balls: Mitigating Bias When Investigating Past Behavior and Future Intent

#dr2021

There are a lot of cognitive biases.

Sometimes they help, sometimes they hurt our decision making.

Vidhika will be talking about the biases that crop up when we ask people to speak about their past and predict the future.

"What people say, what people do, and what they say are entirely different things"

About a decade ago Avataar was released — about the same time companies started getting excited about 3d TV's.

There was a lot of research about people being jazzed about 3d TV's and the companies invested in them...but they kinda flopped.

The research was heavily reliant on people predicting the future and leaning too much on the past — if you ask someone "would you use a 3d tv" they were pumped but the actual experience of watch a 3d TV was very poor.

Just be cause someone liked something in the moment doesn't mean they'll like it down the road.

Our memories are fallible and predictions can be misleading.

Chapter 1: The Pensieve [Harry Potter item where you can relive memories]

People tell us:

What they think we want to hears
What they remember doing
What they're rationalised

So how does it manifest in our research?

Case study on car crashes and leading questions.

Sometimes leading questions don't need to sound leading, it can take one word to lead a conversation.

Loftus and Palmer (1974)

About how fast were the cars going when they:

Smashed/collided/bumped/hit/contacted

Changing that one word made participants respond that the cars were going at different speed.

Following up with another experiment and when they were asked if they saw broken glass in the videos they saw.

32% who were asked if it smashed said there was — there was no broken glass in the video.

Information embedded in our questions can effect their answer.

The Peak End Rule

We have two selfs — the experiencing self and the remembering self.

Even if your experience self changes, your remembering self might stay the same.

The remembering self is roughly an average of how the experience ended and the peaks of the experience

"Odd as it may seem, I am my remembering self, and the experiencing self, who does my living, is like a stranger to me" — Daniel Kahneman

The least likely experience is often the most likely memory

The Hindsight Bias (knew it all along phenomenon)

A lot of times after an event happens, it feels obvious in retrospect.

We transform an event in our heads to be more predictable than it was.

Common during usability testing
- "Oh I totally would have seen that if X"

No matter how much people tended to struggle during a usability test, if I asked them at the end how the experience was they would say "Oh it was really easy I just stuffed up a lot"

Don't over anchor on how people say they experienced something instead focus on the behaviour in the session

Probing

We don't want to probe too much if someone doesn't know something, people may feel pressure to be socially desirable and make up a reason.

We find reasons for the events that happened in our past

When we talk about the past, some other cognitive biases

Confirmation bias
Social desirability bias
Framing bias

There are a lot more but these are common ones we'll come across

Imagining the future is a very recent development in humans evolution.

We're also particularly bad at it.

We can't even accurately predict our own future preferences.

When we ask "would you" it doesn't really mean they're going to do it.

People tend to:

Place a disproportionate weight on the present
Be bad at predicting how we will feel in the future
Overindex on our current preferences persisting

Experiment for 1999

Trying to understand present bias — we make choices today that future is may not like

They had a video list with 20 easy to watch films and 20 dense films

Sleepless in Seattle & Schindlers List was on the list

Half participants were asked to choose a films on the same day

Half participants asked to choose film a day by day

All three on one day: People picked easy films on day 1, heavy films day 2 & 3

Sequential: People who picked day to day chose easy films every day.

Our future selves were aspirational

We do this all the time, like making decisions in the shops about what we're going to eat later in the week (before we order take out).

When we're asking participants about a future state, they're being honest and they're thinking of an aspirational state. By the time the future comes around things may have changed.

We focus on the here and now
We're aspirational about our future
We get more "rational" when forced to wait

People's preferences are time-inconsistent

Actual adoption, actual purchase is likely to differ.

Affective Forecasting

We're not great at predicting our emotions, we can predict positive and negative fairly well, but we're bad at figuring out the intensity and the duration.

We have a psychological immune systems where we bounce back between these highs and lows.

Good and bad events tend to be less intense and last a shorter time than we believe they will

Can sometimes be called Impact Bias

Projection Bias

We underestimate our current and future preferences and values will align.

We project our values and beliefs right now onto future us.

If you've ever gone grocery shopping on an empty stomach you will understand. You can project your hunger state onto your future state.

There will be. a lag between when we do the research and when products come out, the closer and quicker the feedback loops happen the better it is for us.

Try and ask about the near future, anchor it to a time they can picture as close as possible to their current reality.

How can you manage these biases in your work.

Limit Time To Recall

Timely synthesis and debrief
Carefully-considered recruitment windows
In-context thoughts are better than recall

Triangulate

Across teammates
Quantitative and Qualitative data sources
Where are the gaps in the data?

Frame Questions Neutrally

Open-ended questions
Words with subtle influence
Use cognitive testing to make sure the question made sense

Strive For Diversity

On your team
On your participants

Avoid Hypotheticals

Look for observable behaviour
Stay away from "What would you do it..."
Focus on the near future, not the far future

Get Specific About The Past

Focus on the specific past, not the general past
Make a judgement call when to probe further
Are they salient experience or common experiences?

Bring Our The Salt Shaker

Watch out for "i would have done.."
Treat answers to hypotheticals as ideal states
Beware of unwitting exaggerated emotions

Thanks so much @vidhster !!!

#dr2021

@threadreaderapp unroll please