Thread by @stats_q

Afternoon twitter! Some of you might have read some angry tweets from @tmorris_mrc ranting about a paper recently published in Lancet RM, purporting to investigate mechanisms of action of treatment in the FEAST trial, without even mentioning the word “mediation” in the text.[1/n]

The paper had serious flaws both in terms of missing data handling and causal claims. So, we (James Carpenter and I from the @MRCCTU) teamed up with two causal inf. experts (@bldestavola and @richardaemsley) to write a letter to @LancetRespirMed: [2/n]
thelancet.com/journals/lanre…

Background: children with shock are treated with fluid boluses as standard of care in Europe, but this is based on very little evidence. Children generally feel better, and look healthier, after boluses, but no large RCT had ever been done to test the effect on survival... [3/n]

..until ~10 years ago, when FEAST randomised ~3000 children from countries in Africa to one of three groups: two bolus groups and one no bolus group. Surprisingly, the trial found that both bolus arms were worse than the no bolus arm. [4/n]
nejm.org/doi/full/10.10…

FEAST sparked a lot of discussion, because of how unexpected these results were. Remember that European clinicians had recommended boluses for years, and these results went against most of what they had thought about the treatment of children with shock for many years. [5/n]

Several explanations were proposed in a series of letters, but none has been backed up by data yet. It's important to understand WHY boluses did not work in FEAST, to see whether the results were generalisable to resource-rich country settings, where they are still used. [6/n]

It is of course difficult to do research on a treatment considered standard-of-care, as some think it is not ethical to randomise children to no treatment (although one might say that what was unethical was to give treatment without strong evidence for so many years). [7/n]

So re-analysing the huge amount of data that FEAST collected to investigate mechanisms, as Levin et al did, was per se a good idea, but… To investigate mechanisms, you have to do a mediation analysis. [8/n]
[Link to paper] thelancet.com/journals/lanre…

There are several mediation analysis methods,but they decided to use none of these. They fit a Cox model with treatment as the only variable, and found treatment was significantly harmful. They then added other variables, until treatment effect was not significant anymore. [9/n]

Their conclusion was: see, the variables we included explained the harmful treatment effect. Problem: you can do this only with baseline variables. Of course baseline variables cannot explain mechanisms, so you’d do that for other reasons, e.g. to adjust for confounding. [10/n]

They instead added post-randomisation variables to explore mediation. This can go wrong for at least two reasons: the first is that they used hazard rates as estimands. The second is confounding. "Confounding?", I hear you saying. "But this is a RCT!" [11/n]

The problem is that treatment is randomised, but the mediator is not. So the mediator-outcome association is still potentially affected by confounding. No attempt was made to try and identify plausible confounders and adjust for them. [12/n]

Finally, although FEAST collected a lot of information, some of the key variables were not observed at 1 hour, ie after the first bolus was given. That is unfortunate, but nothing we can do about it. In that paper, they decided to impute these missing values, in two ways: [13/n]

1) for a selected group of survivors,these variables were observed at 24 h. So they used interpolation to impute at 1 h. How can we know there was a linear increase between 1 and 24 h, though? Also, trajectories were certainly different for the selected group of survivors. [14/n]

Furthermore, since 24 h was only available for few ids, they imputed assuming the same effect applied to everybody in a specific arm. Because of this, including treatment arm and these imputed values was a bit like including treatment name and ID in the same analysis [15/n]

They are perfectly confounded and not distinguishable. So, no surprise the analysis adjusting for both said the effect of both is less strong. (In the treatment ID/name example, the analysis would simply fail because of perfect collinearity). [16/n]

We could have even imputed a fixed growth in height by 5" for children in the control arm and 10" in the treatment arm, and including height in the model we would have found the same. Still, anyone would agree growth is not a mediator of the effect of bolus on survival. [17/n]

Method 2 involved imputing again a fixed difference by treatment arm,but using values derived from literature,instead of data. Same problems apply,and no need to explain why it is not a good idea to stop observing variables, and start “assuming” what their values would be. [18/n]

The authors responded,saying their analysis was not meant to be interpreted causally.However,they used the word “mechanism” in the paper.There is no mechanism without cause. To put it in @_MiguelHernan words:they are just avoiding to use the C-Word: [16/n]

https://twitter.com/_MiguelHernan/status/978734612875800576

In conclusion: this analysis, as performed, was not needed and does not add anything to the scientific debate. An opinion piece explaining plausible mechanisms and calling for more research in the area would have had the same scientific value, and more integrity. [17/n]

And to conclude, one thing is definitely important, and on that we agree with the authors: we need more research into the effect of fluid boluses for children with shock, to understand whether they are safe or not in different settings from the ones considered in FEAST. [18/n]