Thread by @garjoh_canuck

🤔DO ADS ACTUALLY WORK?🤔
🧵We tackle this question using a large collection of display ad field experiments in a working paper with @EconInformatics & @enub
"The Online Display Ad Effectiveness Funnel & Carryover: Lessons from 432 Field Experiments”
ssrn.com/abstract=27015… 1/8

Recently, the effectiveness of advertising has been called into question. Some of this skepticism is healthy. Measuring the effect of advertising properly is *really* difficult, as I detail here: . 2/

On average, our ad experiments run for 20 days & contain 4M users: 1.6B user-campaign observations in all! Our data come from the 1st months of Google’s Conversion Lift & Brand Lift products, which are powered by our (predicted) ghost ads methodology: doi.org/10.1509/jmr.15… 3/

We measure ad effects on user visits to the advertiser’s website & user conversions: e.g. purchase, sign-up, or download. Our lift estimates compare experimental Treatment users who can see the focal ad & Control users who can not.
% Lift = (Treatment - Control) / Control
4/

We show our site visit estimates (dots) ordered by lift with 95% confidence intervals (whiskers). The median lift is 16.6%. The estimates are noisy, but 85% are positive. Note: we rarely see negative lifts due to industry & academic publication bias. 5/

195 of our 347 site visit estimates are positive & statistically significant (5%, two-sided).
What is the chance that advertising does not work, but we observe 195 of 347 significant estimates?
7.4 * 10^-213. In other words…
6/

We next look at the lift estimates for conversions, which we observe for 184 campaigns. The median lift is 8.1% with 10th & 90th quantiles of -8.9% and 83.4%.
52 of 184 are statistically significant, yielding collective significance of p= 2.9 * 10^-40.
7/

In sum, we contribute some of the strongest evidence that ads actually work. We add to a literature in marketing that seeks to answer this question with multi-advertiser collections of field experiments in various media. FIN