Thread by @rohankatyal29

Working on democratizing experimentation? Here are the resources that I used to 10x volume of experiments at Yelp

1 Good experimentation strategy
2 Focus on people over tech
3 Avoid common gotchas
4 Prioritize for impact
5 Building your own platform
6 Build safeguards

👇

1. Good experimentation strategy

- Furthers understanding of users not just metrics
- Has a strong well reasoned hypothesis
- Ensures you can learn why it didn’t & did work
- Instruments & defines metrics upfront

reforge.com/blog/good-expe… via @far33d @reforge @bbalfour

2. Focus on people over tech

Cultivate curiosity by showcasing potential → Provide practical stats education → Empower with tools → Accept failed experiments & iterate

hbr.org/2020/03/produc… via #StefanThomke

3. Avoid common gotchas

- Define the measurement strategy upfront & get alignment
- Conduct AA tests to spot buggy infra issues
- Factor in outliers while computing the metrics
- Most interesting treatment effect is found in segments

via @ronnyk

4. Prioritize for impact

EVELYN lays out the framework to prioritize experiments based on impact and confidence.

airtable.com/shrInOxl0SL1Xm… via @dariusmc

5. Building your own platform

Under the hood of @Uber experimentation platform → eng.uber.com/xp/

Scaling @Airbnb experimentation platform → medium.com/airbnb-enginee…

6. Build Safeguards

Invest in Guardrail metrics in addition to decision metrics to guard key parts of the business from unwarranted impact by other experiments. Especially important when the volume of experiments starts to go up.

exp-platform.com/Documents/2016… via @Microsoft

The Yelp Story: Building an Experimentation Program From the Ground Up

- Establish baseline
- Integrate technically
- Boost trust in platform
- Faster time to value
- Integrate culturally

vwo.com/webcast/build-…

List of more such awesome experimentation resources → notion.so/Experimentatio… via rohankatyal.com

Back to top of tweet

https://twitter.com/rohankatyal29/status/1308434619918725120?s=20