Thread by @garjoh_canuck

🔥👿🔥 New working paper alert! 🔥👿🔥
"Inferno: A guide to field experiments in online display advertising"
ssrn.com/abstract=35813…

THREAD: This guide reviews challenges & solutions from a decade of research.
#marketingacad #econtwitter #fieldexperiments

“Abandon all hope, ye who enter here” - Dante’s Inferno👿
Online display ad experiments are hell. They are also a proving ground for field experimenters, & have much to teach us. The guide is organized into the nine 9 circles of 🔥hell🔥 as applied to #displayad #fieldexperiments

🔥Circle 1🔥 Display ad effects are so small🤏 that observational methods fail🤦‍♀️. Ad effects explain so little variation in ad outcomes, that they get swamped🌊 by unobserved confounds. Like Dante entering the inferno👿, we resign ourselves to the necessity of experiments.😭😭

🔥Circle 2🔥 Non-compliance is a problem.
ITT estimates are imprecise. Placebo💊 campaigns can recover ATET. BUT, placebos are often invalided because ad delivery algorithms match different types of users to the focal ad vs. placebo ad. New solutions: ghost ads👻 & tag campaigns.

🔥Circle 3🔥 Statistical power🔌 is pure hell👿.
Median sample sizes (Lewis & Rao '15) required to reliably reject differences of:
🔼100% ROI ➡️ 3.3M exposed users 😱😭😱
🔼5% ROI ➡️ 1.3B exposed users 🤬😡🤬

🔥Circle 4🔥 Marginal effect of an ad is complex:
Users choose browsing👩‍💻 intensity which limits ad frequency
Advertiser only control🎛️ spend💰 & frequency caps🧢
🔄Ad frequency affects ad outcomes affects ad frequency🔄
Solution: Impression-level randomization🎲

🔥Circle 5🔥 Identity fragmentation: People are broken into fragmented 🆔 by devices 🖥️📱 & cookies🍪
We only observe fragments🧩 of person’s total ad exposure & outcomes ➡️ ad effect estimates are biased (➕OR ➖) 🤯🤬
Solutions: Person-level 🆔, stratified aggregation

🔥Circle 6🔥 Market spillovers🌊: Ad effect estimates also reflect ads shown to control group🤯.
Solutions:
-When competing campaigns overlap, consider ad effects with & without competitor🦹
-ALWAYS Deactivate own🦸‍♂️ non-experimental campaigns in the same marketplace

🔥Circle 7🔥 Frontier econometrics can help optimize ad incrementality: causal ML🤖, continuous time⏲️ modelling, contextual bandits🎰/Thompson sampling🤹‍♀️.
Destination🥇: Estimate the expected incremental effect of each ad (#attribution), to bid in real time⏲️

🔥Circle 8🔥 Even more problems😢: influential outliers🌠, (incomplete) evolution🦕 of ad metrics🖱️, long run👵 effects of ads have worse power🔌, activity🏃‍♀️ bias, spillovers to household members👨‍👩‍👧‍👧🏡, lack of offline🌍 ad & sales data, & untraceability of cash💵 purchases.

🔥Circle 9🔥 Conclusion: Biologists learn a lot from thermophilic microbes🦠 living in temperatures up to 122°C🥵. In the same way, online display ads👿 provide many lessons for field experimenters👩‍🔬. Research here is rapid, relevant, & exciting!😇