THREAD: How to improve your startup's conversion rate through A/B testing.
We've consolidated learnings from running 1000s of A/B tests for companies like Segment, Microsoft, and Tovala.
We've consolidated learnings from running 1000s of A/B tests for companies like Segment, Microsoft, and Tovala.
A/B testing = the science of testing changes to see if they improve conversion.
This thread covers:
1. Deciding what to A/B test
2. Prioritizing valuable tests
3. Tracking and recording your results
This thread covers:
1. Deciding what to A/B test
2. Prioritizing valuable tests
3. Tracking and recording your results
Testing makes or breaks growth.
We've worked with companies that were failing to convert their traffic.
But after three months of landing page A/B testing, they got traction.
The key: They continuously made their messaging more clear and their offer more compelling
We've worked with companies that were failing to convert their traffic.
But after three months of landing page A/B testing, they got traction.
The key: They continuously made their messaging more clear and their offer more compelling
Every day of the year, a test should be running—or you're letting traffic go to waste.
A/B testing isn't about striving for perfection with each variant. It's about iteration.
A/B testing isn't about striving for perfection with each variant. It's about iteration.
The A/B testing process
1. Decide on and prioritize high-leverage changes
2. Show some % of your visitors the change
3. Run it until you reach a statistically significant sample size
4. Implement changes that improve conversion
5. Log design/results to inform future tests
1. Decide on and prioritize high-leverage changes
2. Show some % of your visitors the change
3. Run it until you reach a statistically significant sample size
4. Implement changes that improve conversion
5. Log design/results to inform future tests
How to source test ideas:
• Survey users. Ask what they love about your product
• Use tools like Hotjar or FullStory to find engagement patterns: What are they clicking vs ignoring?
• Your best ads have value props, text, and imagery that can be repurposed for A/B tests
• Survey users. Ask what they love about your product
• Use tools like Hotjar or FullStory to find engagement patterns: What are they clicking vs ignoring?
• Your best ads have value props, text, and imagery that can be repurposed for A/B tests
Sourcing cont.
• Mine competitors' sites for inspiration. Do they structure their content differently? Do they talk to visitors differently?
• Your support/sales teams interact with customers & know best what appeals to them
• Revisit past A/B tests for new ideas
• Mine competitors' sites for inspiration. Do they structure their content differently? Do they talk to visitors differently?
• Your support/sales teams interact with customers & know best what appeals to them
• Revisit past A/B tests for new ideas
Prioritizing tests:
1. Micro variants are small, quick changes: Changing a CTA button color
2. Macro variants are significant changes: Completely rewriting your landing page
Prioritize macro changes bc they're higher leverage—they often result in large conversion swings.
1. Micro variants are small, quick changes: Changing a CTA button color
2. Macro variants are significant changes: Completely rewriting your landing page
Prioritize macro changes bc they're higher leverage—they often result in large conversion swings.
You'll more often A/B test earlier parts of the funnel—for two reasons:
1. Earlier steps have larger sample sizes—and you need a sufficient sample size to finish a test.
2. It's easier to change ads, pages, and emails than it is down-funnel assets like in-product experience
1. Earlier steps have larger sample sizes—and you need a sufficient sample size to finish a test.
2. It's easier to change ads, pages, and emails than it is down-funnel assets like in-product experience
Other prioritization questions:
• How confident are you the test will succeed?
• If a test succeeds, will it significantly increase conversion?
• How easy it is to implement?
• Is your test similar to an old test that failed?
Start w/ low effort, high-leverage changes
• How confident are you the test will succeed?
• If a test succeeds, will it significantly increase conversion?
• How easy it is to implement?
• Is your test similar to an old test that failed?
Start w/ low effort, high-leverage changes
2 keys to setting up tests:
1. Run one A/B at a time. Otherwise, visitors can criss-cross through multiple tests when changing devices (e.g. mobile to desktop) across sessions.
2. Run A/B variants in parallel. Otherwise, the varying traffic sources will invalidate your results.
1. Run one A/B at a time. Otherwise, visitors can criss-cross through multiple tests when changing devices (e.g. mobile to desktop) across sessions.
2. Run A/B variants in parallel. Otherwise, the varying traffic sources will invalidate your results.
2 tools for running tests:
1. Google Optimize—free A/B testing tool that integrates with Google Analytics and Ads.
2. Optimizely—better flexibility and insights.
We suggest starting with Google Optimize, then getting a demo from Optimizely to see if it's worth the upgrade
1. Google Optimize—free A/B testing tool that integrates with Google Analytics and Ads.
2. Optimizely—better flexibility and insights.
We suggest starting with Google Optimize, then getting a demo from Optimizely to see if it's worth the upgrade
To statistically validate tests, you need:
• 1,000+ visits to validate a 6.3%+ conversion increase
• 10,000+ visits to validate a 2%+ increase
Without lots of traffic, focus on macro > micro variants. Macros can produce 10-20%+ improvements vs micros 1-5% increases.
• 1,000+ visits to validate a 6.3%+ conversion increase
• 10,000+ visits to validate a 2%+ increase
Without lots of traffic, focus on macro > micro variants. Macros can produce 10-20%+ improvements vs micros 1-5% increases.
One note on sample sizes and revenue:
The closer an experiment's conversion objective is to revenue, the more worthwhile it may be to confirm small conversion boosts.
e.g. a 2% improvement in purchase conversion is more impactful than a 2% improvement in "learn more" CTA clicks
The closer an experiment's conversion objective is to revenue, the more worthwhile it may be to confirm small conversion boosts.
e.g. a 2% improvement in purchase conversion is more impactful than a 2% improvement in "learn more" CTA clicks
To track tests, mark the following in a tool like ClickUp:
• Conversion target you're optimizing for: Clicks, views, etc.
• Before & after: Screenshots + descriptions of what's being tested.
• Reasoning: Why is this test worth running? Use your prioritization framework here.
• Conversion target you're optimizing for: Clicks, views, etc.
• Before & after: Screenshots + descriptions of what's being tested.
• Reasoning: Why is this test worth running? Use your prioritization framework here.
When each test is finished, make note of:
• Start & end dates
• Sample size reported by your tool
• Results: The change in conversion, and whether the result was neutral, success, or failure.
If it was a success, note whether the variant was implemented.
• Start & end dates
• Sample size reported by your tool
• Results: The change in conversion, and whether the result was neutral, success, or failure.
If it was a success, note whether the variant was implemented.
Then ask: What can we learn from the test?
1. Use heatmaps to figure out why your variant won. e.g. maybe users were distracted by a misleading CTA in one variant
2. Survey customers who were impacted by the test
Figuring out *why* each variant wins will inform future tests
1. Use heatmaps to figure out why your variant won. e.g. maybe users were distracted by a misleading CTA in one variant
2. Survey customers who were impacted by the test
Figuring out *why* each variant wins will inform future tests
Takeaways:
1. A/B testing is higher-leverage and cheaper than most other marketing initiatives.
2. Focus on macro variants until you run out of bold ideas.
3. Diligently track A/B results and reference them when ideating future ones. Learn from your past mistakes.
1. A/B testing is higher-leverage and cheaper than most other marketing initiatives.
2. Focus on macro variants until you run out of bold ideas.
3. Diligently track A/B results and reference them when ideating future ones. Learn from your past mistakes.
If you found this valuable, give us a follow.
We post threads on all growth channels 1-2x per week (like this one on email marketing):
We post threads on all growth channels 1-2x per week (like this one on email marketing):
https://twitter.com/GrowthTactics/status/1341426684218753024
And if you like our threads, we save our best email insights for our newsletter.
Get bi-weekly growth tactics here: demandcurve.com/newsletter
Get bi-weekly growth tactics here: demandcurve.com/newsletter
• • •
Missing some Tweet in this thread? You can try to
force a refresh
