But, it’s true. I saw this in action recently and wanted to share that story with you.
3/ In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.
Now, that's baffling...
4/ Our team spent a good deal of time trying to understand why was that happening.
Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn’t appealing to them.
Was it true?
5/ When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn’t stop working) and guess what, that automated service used Windows.
6/ After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable.
DUH
🙃
7/ This is a perfect example of Twyman's law.
Remember, if the data is too good to be true, it's probably wrong.
8/ Many extreme results are more likely to be the result of an error in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error.
9/ That's it :)
Hope this mental model was new to you (it certainly was to me!).
If you also have a story related to exaggerated data to share, do reply and share!
• • •
Missing some Tweet in this thread? You can try to
force a refresh
Effective technique for not getting involved with thoughts and emotions.
Correction: it’s acceptance commitment therapy.
By the way, not fusing with thoughts and emotions is the core of mindfulness and is pretty powerful.
Thoughts and emotions that bubble up to your consciousness is mostly clickbait - they got selected precisely because they’re exaggeration’s fabricated to make you pay attention.
The best guidelines for any forum/network I've seen is that from Hacker News.
And the wonderful thing is that these guidelines actually work - Hacker News is the most inspiring and thoughtful forum out there.
Other networks like Twitter can learn a thing or two from it.
Sidenote: the massive work of moderating this long list of guidelines is done by ONE person.
Though increasingly we can have LLMs (like ChatGPT) interpret such guidelines and try to provide feedback to people before they make low-effort, clickbaity, rage-inducing comments.
The guidelines are worth reading in full and internalizing if you want to be a more thoughtful communicator.
• For the non-creator, it appears that the world is falling apart as they see extreme, hot takes all around them as nuanced, well-balanced content is seldom promoted by the algorithm