Thread by @RohanIrvine

Keynote speaker for #designresearch2020 is @leisa, head of design research at @Atlassian

The five dysfunctions of democratised research.

Leisa is acknowledging the traditional custodians of the land she works on, Gadigal of the Eora Nation, the traditional custodians of this land and pay my respects to the Elders both past and present.

Leisa leads the research and insights team at @Atlassian — who make software for teams who collaborate with each other.

"How can we unleash the potential of user research in our organisation?"

Particularly where we run into issues scaling.

Since Jeff Bezos talked about customer obsessed we've move away from fighting for access to customers to how to we cope with everyone wanting to speak to customers.

Some researchers think researchers are the only ones who should be conducting research.

In Australia we don't seem to hire dedicated research professionals.

Leisa did some desktop research about the user research Jobs posted on London last week.

Leads
Seniors

Specialist research roles

San Francisco.

602 jobs

Lead
Managers
Seniors

Again, specialist research roles.

Sydney though.

48

And we're quite liberal on the definition.

Leisa says people who've moved back from overseas to Sydney have been told they would need to do design if they wanted to do research.

So who should talk to customers?

It's become a redundant question.

Today the reality is that all kinds of people believe it is part of their roles to conduct research with customers and users.

More people being closer to users and customers will lead to better products and services.

In order to meet this demand we need to do 2 jobs.

They involve very different skills, knowledge and practices.

Dysfunctional 1: Speed

You know you're working in an engineering driven org when it's 'just known' that Dev work is really the only true work, and everything else before cutting code is largely slowing progress.

The Quest for speed means we're making read offs.

The first quality trade off is in recruitment.

People with relevant experience & context.

Recruitment can be very time consuming.

Unsurprisingly they'll compromise on the recruitment.

Where the trade off comes

1. You relax the recruitment brief
2. You go to online sources where the screening is easier to pass.

The second way you trade of quality is in the analysis and synthesis phase.

When we started we were told 2 hours analysis to 1 hour research.

We are seeing scrapier, lean approaches to analysis where we huddle, make a decision, and move on.

Sometimes we get our participants to pretend to be someone they aren't in a hypothetical situation.

If you have people answering questions that were meant for other people you're likely going to invalidate the information you receive from the participant.

Without taking the time to properly analyse and synthesise your research you are leaving yourself open to bias your outcomes.

It's not to say that you should compromise at all.

It's about being aware and acknowledging what you are trading off and how that impacts your research and the product.

So how can you mitigate the Speed dysfunction?

Dysfunction 2: The Silo Dysfunction.

Ask yourself "Are we getting enough of the picture to be able to make a good decision"

Imagine your team is tasked with building a chair.

What is the chair for? (Lounge, theatre, office, coffee shop, waiting room)

All chairs require a context that without would render them useless.

We need to care about the context, even if we don't own it.

It matters what happens off the screen.

There is a relationship between what happens when people aren't using your products.

People will give us information if we ask

What are the trade offs of the silo dysfunction.

1. Is this an insight my team can act on immediately?

2. Does this give me a definitive answer—we are not interested in something that creates more questions.

You increase the false positives, invalidating your research.

Teams believe they're taking a user Centred, evidence centred approach but being far too narrow and increasing their false positives.

To mitigate this, every conversation has to be centred around "what is the user need that this product is meeting" and it cannot involve your solution/feature in the answer.

Dysfunction 3: Weaponising research.

A few months ago the Ops team were about to implode from the amount of research being requested.

They found a huge amount of hyperfocussed research being commissioned.

The relationship had soured so much that you could only have a discussion about design if you had evidence.

We were trying to win design arguments with graphs and tables, instead of design language.

So how do you build the confidence and trust of design in your organisation?

The most effective tool in mitigating weaponised research is focussing on the end to end journey.

Aligning on the bigger picture helps you have shared priorities and keep us connected to the customer purpose.

At Atlassian teams have to demo the end to end journey, and how the feature they're working on fits into that journey, even if it means demoing work other teams have done.

Dysfunction 4: Quantitative Fallacy.

Lord Kelvin would not have gotten along well with qualitative researchers.

Surveys in a nutshell.

When done well, surveys are great.
But it takes time.

It's more than agreeing on some questions and banging them in a survey monkey form.

Very few people spend a lot of time making sure their questions are great.

In design research we don't seem to practice Cognitive Interviewing, where you interview people to see if they're answering your questions how you expect them, before you send them off to the world.

Many people are much less comfortable with ambiguity than we are.

They are much more comfortable with a big bold number.

How to we get organisations to care about our rich qualitative data?

Roll with the organisation culture, use mixed methods (qual + quant menthods)

Use numbers to open the door to a much more nuanced and deep conversation on the topic.

Dysfunction 5: Failure to thrive.

The important thing about this quote is the context.

Steve is not giving this advice to organisations who have mature practices and resources.

He is giving this to organisations that currently do no research.

Leisa's maturity model.

Customer led

Customer involved

Customer experience led

On the Y axis:
Ignorance
Fear
Enthusiasm
Capability

If we have a high maturity within an organisation we will see both core skills wing looked after

What do you do when you get this objection?

We must be able to match our method to the risk.

What can we do to reduce risk to the organisation.

Leisa has a simple 2x2 to have this conversation in the business.

What does it look like when you plot methods on this 2x2?

There are plenty of ways for scaling and democratising research can go very very wrong but we have the tools in place in our team to make it more effective!

Thanks @leisa!

Audience Q: Do you recommend a place to recruit?

A: We use a range of channels to recruit because it gives you the opportunity to optimise for the channel, and gives you a plan A,B,C if one is not working.

Q: when you already have continuous discovery, how do you convince the business to respect and support you?

A:You have to do things that achieve that for yourself. What are you doing, how are you doing it, what do you need to do to meet that need more effectively.

A: It's incumbent on us to understand what will get our stakeholders attention, get you invited to meeting and asked your opinion on things.

Q: do you have any case studies or cognitive interview examples?
A: don't have any that are publicly available but will put one together soon. You can look online to find it

Q: how do you teach the company to go back to objectives?
A: when I'm working with a large organisation I use a train metaphor, no-one is going to stop for you but you pick a train to jump on, understand what the needs are and try and provoke questions.

A: how do we get them to ask the questions that we want them to ask.
Telling them "these are the questions you need to answer" will not work

Sometime you conduct research that no-one has asked for because you know it will provoke those questions.

Q: how did you transform the NPS?
A: it used to pop up mid task, it was a live number and huge amounts of time were spent trying to figure out why it was bouncing.

We capture it using the happiness tracking survey, it's our of product and we survey via email.

More control over samples, we can ask more questions and people take the time to give us long thought out responses to the open questions.

We also changed the analysis process.

We do monthly update on numbers, do quarterly reports, we get the whole team to do the analysis.

Getting away from matching and counting keywords

Q: how do you advocate for teams with low budget?
A: try to setup a continuous format building your knowledge over time, maybe not super frequently but over time. After some time you will have enough people to do some analysis

Be careful not to use low cost tools

Q: how do you advocate for a seperate design research role in a company?
A: one way, weigh up how long it takes for your designers to do on-top of their job and what risk you're introducing by not having a researcher.

If you really focus on risk, what you're not doing, what it means and what value you'll get from it.

Q: who decides how much knowledge and risk they have?
A: we do it as a team, from a product point of view the buck stops with the product manager.

Again, wonderful presentation, very thought provoking, lots of good advice and fantastic questions from the audience.
#designresearch2020

@threadreaderapp unroll please!