Thread by @samcwl

Some folks may have seen Duolingo's viral TikToks, but not many know about how we think about product development.

Here's how we test and iterate on products at Duolingo 🧵

We take a hypothesis driven approach and Test Everything.

Virtually all of app changes are gated and rolled out using A/B tests.

(and sure, pre-empting the usual: there's a whole can of worms to be opened about hill climbing and local maximas, and would love to talk more about that, but that's for a future thread.)

First, to set context, here's an example of an A/B Test:

Step 1: Have a hypothesis (a falsifiable statement) about user behavior.

We get our hypothesis from "a combination of user feedback, data from our learners, past experiment learnings, and our own intuition".

In the example above, it could be: the word "trial" triggers an alarm in our users' brains, making them less likely to try out the Super experience.

Step 2: Test our hypothesis as rigorously as we can.

Any software makes implicit assumptions and tradeoffs about user behavior everyday.

An *opinionated* product does this especially well.

But, how do we get signal that we are making the right product calls?

Pre-Product Market Fit, we typically do it with deep user research and analytics.

Post-PMF, we do it with A/B tests at scale.

Step 3: Analyze and Call the Experiment

All product development have tradeoffs.

These can be pre-empted and discussed in the design phase.

There's sometimes a tradeoff between user retention and monetization, or monetization and learning, or user retention and learning.

But these can also be unexpected:

We have guardrails to ensure that these don't happen.

Sometimes teams can foresee this, while other times they can't and they have to dig deeper.

Step 4: Document your learnings and socialize them.

What did you learn about user behavior? Anything unexpected? What are the key insights?

What are the next steps?

Bringing it home: our purchase flow.

You can see the evolution in our thinking here.

Over the years, you can see:

Hypotheses of...

testing clear upfront pricing,

reducing choice,
calling out user benefits,

testing redesigns (both of Duo and the app!)

(shoutout to our tireless Monetization Area's work on this)

If interested, read more here in our latest shareholder letter and listen to the shareholder call with @LuisvonAhn and Matt Skaruppa.

investors.duolingo.com

Also see more examples of past experiments on our blog

(blog.duolingo.com/how-duolingo-s…)

Future threads coming about how our experimentation team built this, what we learned along the way, and where we are going.

Oh and Duolingo is still hiring!

Come work with folks who are the best in the business @cemkansu @Albertc248 @nickeyskarstad and many more!

careers.duolingo.com

You can read the unrolled version of this thread here: typefully.com/samcwl/qER6kOQ