Profile picture
Ellie Murray @EpiEllie
, 21 tweets, 11 min read Read on Twitter
Have you heard of #PARACHUTETrial?

@rwyeh & his team convinced 23 adventurous people to jump out of a plane wearing either a parachute or an empty backpack!

This trial is a great chance to talk about target validity: what are we asking & for why? bmj.com/content/363/bm…
First let me say I’m not an expert on target validity, because it’s a new concept. But I think this trial is a nice case study for it.

Check out the great paper introducing the concept, here: doi.org/10.1093/aje/kw…
So, what is target validity? When we do an #RCT, we want to know about the effect of a treatment *in a target population*.

Target validity tells us about how well we have learned about that from our trial (or observational study).
In the #PARACHUTEtrial, we want to know:

What is the causal effect of using a parachute on 5-minute post-jump survival among the population of people who are about to jump out of an airplane, vs an empty backpack?
We can split target validity into two main components: internal validity and external validity.
Internal validity means the estimate we got *in our study sample* is the right answer (or close to it) for *that group* of people.

In the #PARACHUTETrial, internal validity depends on whether we have a good estimate of the effect of using parachute vs backpack *in our trial*
That is the per-protocol effect: the effect of parachute vs backpack if everyone had adhered to their assigned treatment.

#PARACHUTETrial had perfect adherence & complete follow-up, so this is easy to estimate: the per-protocol effect is *equal* to the intention-to-treat effect
In the #PARACHUTETrial, the intention-to-treat effect (and thus the per-protocol effect) is:

P(death=1|assigned parachute)=0/12
P(death=1|assigned backpack)=0/11

0/12 - 0/11 =0 percentage point risk difference of death at 5 minutes post-jump.
Our estimated intention-to-treat and per-protocol effects have great internal validity.

There’s no confounding and there’s no loss to follow-up, so this is probably the true value *for the study population* (although, as others have noted the sample size is quite small)
But we’re not done! The next part of target validity is external validity.

This asks us to assess whether the answer we got for our study population tells us anything about our target population!
The target pop might be everyone with a particular diagnosis, or adults over 40, or those who have failed first line therapy.

In the #PARACHUTETrial, the target population is anyone about to jump out of any airplane.
This is where #PARACHUTETrial falls down.

They tried to recruit people currently on an airplane, but only succeeded in recruiting people who got onto a stationary grounded plane for the purpose of becoming eligible for the trial!

That’s different from the target pop!
Since the study sample and the target pop are different, we need to think about whether those differences affect our interpretation of the results.

There are 2 things to consider:

1️⃣is the effect the same for everyone?

2️⃣if it’s not, are types of people diff in these 2 pops?
The main diff bt/wn #PARACHUTETrial study & target pops is height & speed of plane they’re planning to jump from.

1️⃣Do parachutes work same no matter speed & height of plane? No!! We definitely don’t think so!

2️⃣Do the types of people in 2 pops differ? Yes! We just said so!
Because plane height and speed differ in the trial & target pops, & because we think parachutes work differently at different plane heights/ speeds (ie plane height and speed are “effect modifiers”), the #PARACHUTETrial has low external validity *for our target pop*.
(That doesn’t mean it has low external validity for *all possible target pops* — it’s probably externally valid to a target pop comprised of all people planning to jump from stationary grounded planes for example)
Returning to our original question:

Does #PARACHUTETrial tell us about the effect of a parachute (vs backpack) on 5-minute post-jump survival among people who are about to jump out of any airplane?

No! Target validity for this question is low!
Let me just finish by saying that none of this is a criticism of #PARACHUTETrial — the authors describe the low external validity very clearly in their paper.

Which is what makes it such a good case study.
Lastly, I’m still learning about target validity, so I hope the authors of the target validity paper will chime in if I’ve misrepresented or misunderstood their concept!

@Lizstuartdc @leskocar @danielwestreich @edwardsjk
A counterpoint: perhaps the #PARACHUTETrial actually had 0% adherence ... everyone *wore* their assigned gear, but no one actually deployed their parachute! See this article & thread 👇🏼
Missing some Tweet in this thread?
You can try to force a refresh.

Like this thread? Get email updates or save it to PDF!

Subscribe to Ellie Murray
Profile picture

Get real-time email alerts when new unrolls are available from this author!

This content may be removed anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just three indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member and get exclusive features!

Premium member ($30.00/year)

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!