Thread by @joulee

My co-founder Chandra Narayanan's quote has become something of a product-builder's mantra for us: Diagnose with data and treat with design.

There is so much packed into those sentences! Thread going deeper 👇 (1/15)

First: "diagnose with data." The job of data is to help you understand the ground truth of what is going on (with your product, user behavior, the market, etc.)

Typically, we humans run on intuition, a rudimentary kind of pattern-matching. This is insufficient in many cases.

Intuition works if you've studied something deeply (think Serena playing tennis.) But it does not serve you well in:

1) Making decisions for contexts you don't understand
2) Generalizing predictions at huge scale / complexity
3) Optimizing the impact of many tiny decisions

When I say "data," I mean objective facts that help you understand people's reactions to what you are building. This can be:

1) qualitative observations (yes, user research or customer discovery is data!)
2) quantitive behavioral data (clicks, views, etc)
3) market data

When you are data-informed, it means that you are paying attention to data points beyond your intuition that give you an understanding of what is *actually* happening, whether problems or opportunities, so that you can make the best decisions.

When people say they want to be "data-driven" with product decision, I get nervous. There are many shortfalls of data, including:

1) Data can tell you what is happening, but not what to do about it
2) Quantitative behavioral data can tell you *what* someone is doing, not *why*

"Treat with design" means that once you understand *what* is happening in detail--what is the problem? What's possible (from benchmarks)? Where are the opportunities?--you can craft a solution.

Design is creative, open-ended problem-solving.

Design is not "the way it looks," or beautiful colors and animations. It's not the brand or the logo.

It is how the product *works*. Designing is the the process of exploring and arriving at a solution. I believe all builders are designers.

Design and data are not at odds with one another. One helps you understand phenomena and gives you a foundation on which to build your assumptions.

The other is the joyful process of creation to solve problems based on those assumptions.

Of course, data and "design intuition" can point at different conclusions. This is usually because:

1) The data is being interpreted incorrectly
2) Design intuition is wrong

How do you know which? My rules of thumb below.

An example of data being interpreted incorrectly: "This change results in a higher action rate, so that means it's better."

Are you sure? Did you check how many people undid their action or pressed "back" after the fact? Maybe they did something they didn't mean.

If design intuition tells you that some experience is bad (because it's hard to use, it's confusing, etc.), TRUST the intuition.

It probably *is* bad for some group of people.

The job now is to understand: for whom, and how bad?

After you learn that, you might still decide to ship the experience (maybe because those people aren't the target customer.)

But you should be informed in making the decision.

If design intuition tells you that A works better than B at a large scale, be wary.

Ex: I like myself a clean, modern interface. I like white space. I prefer the elegance of icons only over icons + text.

For a product with hundreds of millions of users, this intuition is wrong.

The more your target audience does not look like you, the more you should be skeptical of your design intuition. Despite having seen thousands of a/b test results for minor design details, I am still regularly surprised by the results.

In general, use quantitative behavioral data (segment analysis, A/B testing, etc.) when you're optimizing and growing something.

Do customer discovery / user research when you are in the early 0->1 phase.

In conclusion, data helps you become a better designer. But data by itself does not lead to wonderful things. You still have to design them.

Fin.