Thread by @paraschopra

You've heard of AI. But have you ever heard of IA?

🚀🚀

Today, at @VWO, we're announcing a big shakeup of the A/B testing industry.

(a thread about our breakthrough innovation)

1/ Our mission to help marketing and product teams reduce the effort required for figuring what works best for their business

In 2010 we pioneered the DIY visual editor for business teams for editing webpages and creating their variations for A/B tests without involving IT teams

2/ That innovation cut the effort to launch an A/B test from weeks to hours

But, as anyone who has run an A/B test knows, you still have to wait for weeks in order to start getting statistical significant results about which version is better.

Can we cut that wait time too?

3/ Wouldn’t it be wonderful if there was a way to instantly know which version is going to win without waiting weeks or months for the data to arrive?

4/ The first research direction we turned to was the booming AI technology.

If neural networks can drive cars today, perhaps they can also predict which website version is better?

coursera.org/lecture/machin…

5/ Over the last few years, we tried several techniques.

We made some progress when a couple of months ago, we launched an AI-powered website copywriter that helped marketers come up with new alternative headlines, CTAs and product descriptions to test.

vwo.com/blog/ab-testin…

6/ But, unfortunately, similar AI-based techniques didn’t perform satisfactorily on prediction of A/B test results.

It seems driving cars, beating world Go champion and detecting tumors is an easier problem than predicting which website design is going to be a better choice.

7/ IA comes to rescue.

As we were about to almost give up on our dream project, someone pointed us to the elephant in the room: astrology.

8/ Our first reaction was to dismiss this absurd suggestion immediately.

We’re a skeptical bunch and there’s no place for such pseudoscience in a serious business like ours.

9/ But we were desperate and decided to give this totally absurd idea one last shot.

We were told that millions of people across the world predict all sorts of phenomena using techniques like horoscopes, or Tarot cards.

10/ If people are using a technique to predict whether their marriage will succeed or not, can’t it be used to predict whether A version is better than the B version?

11/ After looking at the cost-benefit equation, we decided to do a small pilot.

If, as we predicted, astrology didn’t work we’ll just lose some investment and time.

However, if it did work, it’ll change the marketing and user research industry forever.

12/ So, in all earnestness we kickstarted the pilot.

After doing initial research on the most predictive astrology technique, we settled on Indian Astrology (IA).

Particularly, we settled on parrot astrology.

13/ What gave us confidence in it was that there’s even a wikipedia page on it.

en.wikipedia.org/wiki/Parrot_as…

14/ In parrot astrology, an astrologer lays out multiple options in front of a caged pet parrot.

When the parrot is let out of the cage, it picks one of the options as the prediction.

Simple and straightforward. We loved it already.

15/ Our pilot:

- Take 100 A/B tests from our database where there was a clear winner (either A or B)

- Print their screenshots

- Recruit an Indian astrologer with a parrot and have the parrot pick between the pairs of screenshots across 100 trials (one for each A/B test)

16/ Our null hypothesis was that the parrot wouldn’t be able to pick better than the random chance.

That is, the parrot should pick the correct winner and the loser each roughly with a 50% chance.

16/ 🤯

However, to our astonishment, the parrot picked the winner 80 times out of 100.

Using any standard binomial calculator (either frequentist or bayesian), you’ll quickly find out that the chances of that happening are close to nil.

Indian Astrology really worked!

17/ We’re obviously onto something here.

We soon plan to write a scientific paper on this and submit our findings to a top journal.

It’s all very exciting and mysterious.

How does the parrot know which version works best?

18/ Well, we don’t know that but it shouldn’t prevent us from using the technique in the real world.

Nobody also knows how deep learning works but everyone uses it.

technologyreview.com/2017/04/11/511…

19/ Get early access to parrot-powered A/B test prediction.

We’re actively trying to scale the pilot with an eventual goal for making it available for all businesses in the world via simple user interface inside VWO or via an API.

20/ The main challenge is in finding enough parrots and making sure we’re doing this ethically.

Once we do that, any business in the world will be able to rapidly converge on optimal user interface without the need to do any A/B testing.

21/ We have some slots for people who want early access to this technology.

If you want to give it a shot, email us here: [e-mail]

22/ Wait, what?

Well, didn’t we say we’re a skeptical bunch?

It turns out there’s no scientific evidence that astrology works. en.wikipedia.org/wiki/Astrology…

23/ Moreover, we strongly condemn caging of any bird or animal for anything other than clinical or medical research (even in clinical research, we wish computational and physical models increasingly replace actual animals).

24/ If you were excited about the prospect of prediction of A/B test results, you’re not alone

Even though parrot-astrology doesn’t work, we’re committed to continue making progress towards helping business teams achieve a higher probability of success in each of their A/B tests

25/ After launching an AI-powered website copywriter last year, we recently launched an AI-powered heatmap predictor.

vwo.com/tools/ai-based…

You enter your website URL and it predicts where you'll get clicks on it (we use deep learning for this one)

26/ I'm sorry for giving you hope that you can get rid of A/B testing, but hope you had fun reading the thread :)

Happy April Fools day!

vwo.com/blog/predict-a…