My son just had his 2nd bone marrow transplant. He's receiving treatment for a rare immunodeficiency. While going through this, my wife @IsaGutierrezR and I made a review of the literature. We were shocked on how limited the statistical analysis was in most of the cases 1/n
We are aware that these studies face many constraints: (i) very small sample sizes (many case studies); (ii) no opportunities for a randomized experiment. However, all studies trying to identify casual relations don't even try to meet the most basic statistical conditions 2/n
There's a common belief that economics developed sophisticated statistical tools for causal inference due to the lack of opportunities to experiment in real world settings, and that experimenting in medicine was easier. Well, this is not always true 3/n
We believe it is urgent for researchers in medicine to use better tools, and to move the bar upwards (a lot) in terms of statistical analysis. There should be ways to bring @causalinf, @Adcamach, @RaquelBernal3, @rglenner and many others to help researchers in medicine 4/n
In case anyone wants to replicate this, the name of the disease is MCH Class II Deficiency, or Bare Lymphocyte Syndrome. We found ~30 papers, many included 10+ patients. The largest included 30-35 patients. Next are some of the situations we found 5/n
The most basic: (1) Few studies included complete summary statistics. (2) No study included a regression table. (3) No study discussed how standard errors were estimated. (4) No study published the data (let alone the code) for replication 6/n
The less basic: (5) Many studies compared means between samples with major selection problems (e.g. "controls" were treated 10 years before "treatment" which entails a major lag in medical technology), and didn't even discuss these selection problems 7/n
(6) These studies usually compare means unconditionally (and mention a univariate regression) and report a p-value; then compare means conditioning on observables (and mention multivariate regression) and report a new p-value. No discussion on the variables is reported 8/n
(7) Some studies go beyond the comparison of means between very selected samples, and perform heterogeneity analysis with very small samples of 5-10 patients (with one group treated 5-10 years before the other) 9/n
The most concerning: (9) After doing the analysis, these studies conclude with a level of certainty that is definitely far from realistic, recommending to treat patients one way or another. Next is an example (I will omit the specific reference) 10/n
A study with 33 patients, 12 received treatment A and 21 received treatment B. There is a difference of about 5 years in the timing of transplant between the two samples. A sub-sample was transplanted using cord blood units as the source of stem cells 11/n
This sub-sample includes 7 patients receiving treatment A and 5 receiving treatment B. The study concludes that "the use of cord blood is a risk factor for graft failure..." because the p-value of that mean difference was 0.015. Surely, they are overstating precision 12/n
It's urgent that medical research adopts some of the statistical tools developed in economics and other fields such as political science, and it's also urgent for this research to be more transparent on what is actually being done 13/n
For those interested, our baby is well. This second transplant seems to be working, and we're hopeful the transplant will be curative. We have a long way to go, but these first steps have been successful 14/14
A final important remark: I did not intend to overgeneralize. There’s indeed very rigorous research in medicine, but these limitations are worth highlighting due to the implications they have
Yet a couple of additional things: (1) Erratum (a typo): the name of the disease is MHC (not MCH) class II deficiency. (2) I found many great case studies that didn’t overgeneralize results, these are extremely valuable, especially for rare diseases
