Thread by @willcritchlow

OK. Here we go - thread of answers to questions that came up during my #FOS19 presentation in Amsterdam today - about SEO / CRO / full funnel testing (read more here: distilled.net/resources/anno… ) cc @basvandenbeld

Q: how do you test the homepage of a website?

A: although you can run CRO tests on a homepage, SEO (and hence full funnel) tests require a site section with multiple pages with similar template. You can only really do before/after tests. [contd]

The techniques I described are mainly applicable to large websites with large site sections (e.g. ecommerce, real estate, travel, jobs, large brick+mortar chains etc). In these cases, most organic traffic is not to the homepage

Q: do the tests only commence after Google has crawled and indexed the change?

A: obviously no uplift will be detected until this has happened, we attack this currently with statistics - essentially, you can't get a statistically significant uplift before reindexation [contd]

(and going long enough without an uplift means we can assume reindexation has happened).

We have plans to experiment with more powerful techniques to model this better and / or track recrawl, but this is how it works at the moment.

Q: how do you test on low traffic pages?

A: our approach requires > 1,000 organic visits / day to the site section in question. Below that, as with CRO, you are aiming less for statistical significance, and more for qualitative / subjective outcomes.

Ideas: track rankings instead of traffic, look for traffic uplifts even if not statistically significant, run user trials to determine things like CTR of new titles etc.

Q: does Distilled ODN (odn.distilled.net) replace other CRO tools like VWO?

A: it can replace the need for other tools for testing changes to whole site sections, but doesn't replace the need for classical CRO/UX testing on single pages (e.g. homepage) or e.g. in checkout

Q: do you make combined hypotheses for CRO & SEO at once? How do you isolate effects?

A: we tend to come up with tests from both "directions" [contd]

sometimes in the mindset of having a UX hypothesis and testing search impact as well, sometimes in the mindset of having SEO hypothesis and testing CR impact.

We can isolate effects quite easily - our proprietary statistical analysis focuses on SEO impact and...

...the CRO impact is measured separately in analytics

Q: are B variants from a WYSIWYG editor properly indexed?

A: if I've understood correctly, this is about "traditional" CRO tools and whether their tests are indexed. In general, you want to avoid having that happen - keeping googlebot in your control group until you roll out

Q: do you need to run a CRO test in a dedicated CRO tool *and* in Distilled ODN (odn.distilled.net)?

A: in general, no. If you run a full funnel test you get data on combined performance (net performance) *and* separate SEO and CRO results as well.

That's all for now - about to board my flight - but thanks for having me #FOS19. I had a great (if short) time in Amsterdam. See you all soon. Feel free to tweet more questions at me.