Thread by @shoshievass

Very excited to see my first (and I expect, not last) paper w @ZiYangKang out in print.

Thread👇on what this paper is about + why I hope lots of folks will use it.

This paper is for anyone who does an exercise like this:

1) take data on demand vis a vis a price shock
2) estimate a treatment effect
3) calculate consumer surplus to assess the policy.

The question: how sensitive are your welfare conclusions to functional form assumptions?

Our paper starts at point (3).

Your treatment effect tells you a range of possible consumer surplus estimates – e.g., consumers value a subsidy at least 0 and at most (# subsidized sales x subsidy). But that can be a wide range.

A common approach is to use a simple demand model (maybe the same one used to estimate the treatment effect) – e.g. log(q) ~ log(p) – and integrate under that curve.
But how do you make sure your results aren't driven by functional form? 🤔

Our solution has 3 steps:
1⃣ Start with your preferred demand model (e.g. linear) 2⃣ Ask: how much can demand deviate before conclusions flip?
3⃣ Get a robustness measure: larger deviations needed = more robust results

It's easiest to show how it works through examples.

We work through a simple simulated example in Sec 1 and applied examples from published papers in Sec 4.

Semi-updated slides here: tinyurl.com/kvslides

Suppose we have price/quantity data from a price hike experiment in a few mkts.

The underlying demand curves could be complicated but we only see 2 price points / mkt so we can't do much curve fitting. But we can use the experimental variation to get an ATT of p on q.

We can interpret the ATT as an avg gradient of the demand curve(s).

If we assume demand comes from a simple functional form (e.g. linear/isoelastic/etc.) that's enough to fit the whole curve and compute consumer surplus.

But the CS estimate will vary based on what we choose

How do we check all the possible functional forms that are consistent w our data?

We propose 2 ways, motivated by regression-based functional forms.

A (generalized) linear fn has (transformed) constant gradient + zero curvature. We propose 1D relaxations of each of these.

Each relaxation covers a lot of potentially complicated demand curves: e.g. gradient can vary in any way so long as it's bounded in a range around the avg.

But we don't need to fit any of them; instead we compute bounds on how big/small the CS could be under any curve in the set

Why is this useful?

Say you want to evaluate CS loss against an externality, G.

Under a linear model, if CS loss is below G, then the policy is net good; ow net bad.

We ask: how non-linear would true demand need to be for the result to flip? You can read it off the graph.

This isn't just for linear benchmarks or price experiments.

It works just as easily for any benchmark of the form A(q) ~ B(p) for monotonic fns A(.) and B(.) given a baseline price, quantity and an identified and (somehow) estimated treatment effect of p on q.

Our replication code includes a little python pkg to compute these graphs (and their associated robustness measures) for any benchmark in this family.

For the easy/common cases, we have a table with closed form formulas.

aeaweb.org/articles?id=10…

But wait, why is this useful?

Two thoughts:
1. Suppose you're evaluating a policy like this, estimated the ATT & want to say sth robust about welfare.
Our paper says don't think too hard about how to impute the demand curve; just do what's natural and check how important it is

2. How does this robustness measure tell us how important the demand curve is?

Our measure is a threshold "r" from 0 (not at all robust) to 1 (super robust). If you're close to 1 that's good. If not, "r" tells you conditions on feasible gradients/curvatures that are testable.

Bonus: my thread highlighted the framework/results of the paper, but the meat of why it works rests in a serendipitous (to us, at least) connection with information design that lets us borrow some neat tools that have been growing in popularity in theory but not so much outside.

Tl;dr: Please read the paper. If you want to try to use this in your own work and have questions, don't be shy :) shoshanavasserman.com/rmwa/

@threadreaderapp unroll