Thread by @AmJEpi

It’s time for some #cartoonepi & #epiquiz fun!

Today, let’s talk about Difference-in-Difference analyses and how to use them to estimate the impact of policy changes!

Our example paper is from our May issue by @DrRitaHamad & colleagues.

#epiellie

I love this paper by @DrRitaHamad which tries to answer the question: did updating the allowed contents of the WIC package to include healthier options actually impact diet & nutrition during pregnancy?

Link👇🏼academic.oup.com/aje/article-ab…

First, some background for those of you not familiar with the WIC program.

WIC stands for “Special Supplemental Nutrition Program for Women, Infants, and Children”, and provides vouchers for specific food combos (‘packets’) for low-income pregnant women & kids <5yo in the US.

I learned from this paper that >25% of pregnant or post-partum US mums receive WIC benefits.

That’s amazing, especially given that in their study population only 50-60% of eligible women actually received WIC.

Does that imply up to half of all US pregnancies qualify?!!

Okay, WIC is super important so it must be constantly evaluated & improved right?

Not so fast!

The “packets” were only updated in 2009 after having been the same for 30 years!

And only one study had looked at the impact of the new packets since then—with no control group!

Which is where @DrRitaHamad and colleagues come in. They wanted to know whether the update was helpful or not in actually improving nutrition during pregnancy.

This is not an easy question to answer at all! But it’s *so* important!

What makes this question hard?

Let’s take a quick poll to see what you think and then I’ll tell you what I think. No peeking tho!

(Click tweet to see poll options)

Trick question, they all make it hard!!

WIC participation & nutrition is likely very confounded, not only because you have to be eligible but because getting signed up isn’t easy!

Plus, what is diet or nutrition even? Measuring that well is hard so how do we know if it changes?

But IMO the toughest problem is the control group. We want to know if the new packet improves diet, but compared to what? The old packet in the past? No packet now? The old packet now if the rules hadn’t changed?

Really we want to know that last one, but no one is in that group!

So we have to use some other group to make guesses about what people would have eaten after the rule change *if the rules hadn’t changed*.

This is where @DrRitaHamad & colleagues had the great idea to use the Difference-in-Difference (DiD) method.

#DiD is not at all common in #epidemiology, so let’s walk through how it works.

First, we need to assume that *if the 2009 change hadn’t been made policy* then the diet of WIC recipients wouldn’t have changed any differently from the general population’s diet over the study.

This is the crucial assumption for Difference-in-Difference. If you believe this assumption then everything else is pretty straight-forward.

So, do u think diet would have changed *differently* for WIC recepients vs non-recipients without new packet?

(Click to see poll options)

Let’s look at an example.
In the #epigif below suppose the purple line is diet among non-WIC recepients (including those eligible & not eligible) measured by the Healthy Eating Index.

Then suppose the blue line is the diet among WIC recipient. It may have changed slope in 2009.

What we have to assume for Difference-in-Difference is that *had we not made the policy change* the blue line would have instead kept the same slope, like the dashed blue line does in this second #epigif👇🏼

Okay, so let’s suppose we’re happy with this assumption. How do we use it to figure out the impact of the WIC packet changes?

Difference-in-difference is actually pretty simple!

We look at the difference between the WIC & no WIC groups before 2009, the change in the no WIC group over time, and then the difference between what we *expect* for WIC and what we *actually see* for WIC after 2009, under our assumption.

Here’s an #epigif on how it works

Okay, let’s sum it all up! The #cartoonepi version below shows what this might look like for our question (note: the lines are made up by me—the paper had no graph!)

I’ve added the regression equation & labeled the differences to match the coefficients.

Our target is beta1👇🏼

So, one last thing I wanted to think about is what does the directed acyclic graph (DAG) look like here? I’ve never seen one for DiD before. Have any of you?

The best I could come up with is that we really have 2 graphs: the one we want to assess and the one we actually assess.

In the target DAG, we want to know how WIC contents impacts nutrition. But we don’t have a control group, so we can’t estimate the red arrow directly.

So, instead we collapse the WIC arrows and look at whether the total effect of WIC status is *modified* by date of pregnancy (pre or post 2009).

I think with DiD we are estimating this👇🏼

Our assumption of no other trends in nutrition is (I think) shown in the DAG by the *lack* of arrows from WIC status to nutrition in the target DAG.

What do you think? Anyone from #econtwitter care to comment on these proposed #DiD DAGs?

So, that’s a wrap on Difference-in-Difference!

Anyone got anything to add?

Thinking about using it in your research?

Got any questions or concerns there?

Share your thoughts in the comments!