Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Noah Haber Profile picture
Noah Haber
@NoahHaber
econ, epi, stats, meta, causal inference mutant scientist, epistemic humility fairy godmother, chaos muppet. doing metasciencey stuff, e-mail noah@cos.io
Mar 1, 2022 • 23 tweets • 10 min read
"DAG With Omitted Objects Displayed (DAGWOOD)" is out in Annals of Epi!

What is DAGWOOD?

A framework?
A method for revealing and reviewing causal inference model assumptions?
A tool for model building?
A statement on epistemic humility?

Answer: Yes.

doi.org/10.1016/j.anne… This weird paper could only be brought to you by the weird collective minds of @anecdatally, @SarahWieten, @BreskinEpi, and I.

But before I run through it, an acknowledgement:

It's March 1, 2022, and events in Ukraine and across the globe far overshadow any celebration here.
Feb 21, 2022 • 21 tweets • 5 min read
Time to make it official: short of some unbelievably unlikely circumstances, my academic career is over.

I have officially quit/failed/torpedoed/given up hope on/been failed by the academic system and a career within it. To be honest, I am angry about it, and have been for years. Enough so that I took a moonshot a few years ago to do something different that might change things or fail trying, publicly.

I could afford to fail since I have unusually awesome outside options.

And here we are.
Aug 30, 2021 • 27 tweets • 8 min read
Causal language study is now up on medRxiv!

medrxiv.org/content/10.110…

Ever wondered what words are commonly used to link exposures and outcomes in health/med/epi studies? How strongly language implies causality? How strongly studies hint at causality in other ways?

READ ON! Health/med/epi studies commonly avoid using "causal" language for non-RCTs to link exposures and outcomes, under the assumption that ""non-causal"" language is more ""careful.""

But this gets murky, particularly if we want to inform causal q's but use "non-causal" language.
Aug 17, 2021 • 6 tweets • 2 min read
I've done a fair bit of generating simulated data for teaching exercises, methodological demonstrations, etc.

It's really, really hard to make simulated data look "real," and it usually doesn't take much to see it.

That pops up in a lot of these cases. Granted, we only see the ones that get caught, so "better" frauds are harder to see.

But I think people don't appreciate just how hard it is to make simulated data that don't have an obvious tell, usually because somethig is "too clean" (e.g. the uniform distribution here).
Aug 16, 2021 • 4 tweets • 1 min read
Perpetual reminder: cases going up when there are NPIs (e.g. stay at home orders) in place generally does not tell us much about the impact of the NPIs.

Lots of folks out there making claims based on reading tea leaves from this kind of data and shallow analysis; be careful. What we want to know is what would have happened if the NPIs were not there. That's EXTREMELY tricky.

How tricky? Well, we would usually expect case/hospitalizations/deaths to have an upward trajectory *even if when the NPIs are extremely effective at preventing those outcomes.*
Jul 22, 2021 • 6 tweets • 1 min read
The resistance to teaching regression discontinuity as a standard method in epi continues to be baffling. I can't think of a field for which RDD is a more obviously good fit than epi/medicine.

It's honestly a MUCH better fit for epi and medicine than econ, since healthcare and medicine are just absolutely crawling with arbitrary threshold-based decision metrics.
Jul 20, 2021 • 5 tweets • 2 min read
My R code mixes and matches base and tidy and includes variously apply functions and for loops.

Bring out your pitchfoRks, #RStats. for (i in 1:infinity){

print ("For loops are perfectly fine in R and can be just as efficient as apply in the right circumstances, and are usually the BEST option for operations that rely on previous iterations.")

}
Jul 19, 2021 • 5 tweets • 1 min read
The speed and volume of COVID papers was and continues to be a shameful disaster.

But rather than opportunism; the bigger problem is a system that takes well-meaning people "helping" in ways that are, by and large, worse than doing nothing at all. Ain't just researchers. Back at the beginning of the pandemic, I was very into a maker/hackerspace.

Huge outpouring of well-meaning "helping" by doing things like 3d printing ventilators and whatnot.

But little doing the hard homework of seeing if they were actually useful.
Jul 17, 2021 • 4 tweets • 2 min read
your new favorite dinosaur is the gigantoraptor.

you're welcome.

en.wikipedia.org/wiki/Gigantora… Art credit to cisiopurple h... art credit to cisiopurple at deviantart and shout out in particular to the choice of human model for scale
deviantart.com/cisiopurple/ar…
Jun 7, 2021 • 84 tweets • 19 min read
Status update: protocol is getting close to done, protocol coauthor team finalized, reviewers are being recruited and we're having one of several intro meetings tomorrow morning.

Good thing I definitely for sure planned ahead and made slides.

docs.google.com/presentation/d… Hypertransparency part 2:

The.

Entire.

Project.

Folder.

drive.google.com/drive/folders/…
Jun 7, 2021 • 5 tweets • 1 min read
Periodic reminder that our current peer review *institutions* != the whole concept of peer review.

Yes, our *current peer review institutions* are an utter disaster, but that implies exceedingly little about other implementations of peer review. If I had to guess, the vast majority of claims that "peer review doesn't work" fail to separate implementation from concept and potential.

Any institution that is underemphasized, underfunded, hobbled, opaque, arbitrary, and easily manipulated tends to produce bad results.
Apr 21, 2021 • 65 tweets • 16 min read
New project on causal language and claims, and I want you to see how everything goes down live, to a mind-boggling level of transparency.

That includes live public links to all the major documents as they are being written, live discussion on major decisions, etc.

HERE WE GO! Worth noting: this is the second time I've tried this kind of public transparency; the previous paper got canned due to COVID-related things.

NEW STUDY TIME!

Here's the idea (at the moment, anyway): health research has a very complicated relationship with "causal" language.
Mar 20, 2021 • 24 tweets • 5 min read
Systematic reviews and meta-analyses are like plywood. While they often have a pretty veneer, they are only as useful as the layers of materials they are made of and how it's all put together.

In this essay I will Hm, might do this, since plywood is so, so much cooler than people give it credit for, and there's some good analogy making with how cross-grain layers are complementary and hold things in check.

I have nerd sniped myself.
Mar 19, 2021 • 6 tweets • 2 min read
FWIW, having "grown up" in econ (and now spending 90% of my time in a different field entirely), this statement strikes me as a pretty accurate description of economists as a whole, and a major source of inter-field friction. I do think that there is something to the fungibility of a lot of econ-style frameworks and ways of approaching problems, BUT in combination with hyperconfidence it gets econs (including me) in trouble.

I've had to learn to unlearn a lot of that hyperconfidence.
Mar 19, 2021 • 9 tweets • 3 min read
Now that everyone is (justifiably) up in arms about CurateScience, may I turn your attention to SciScore™, claimed to be "the best methods review tool for scientific articles." (direct quote, plastered all over their website)

sciscore.com For starters, for a "tool" all about transparency, reproducibility, etc., it's pretty damn hard to find what it is they are actually reviewing for.

There is no page explaining it, nothing in the FAQ, and you have to dig around a bit.
Mar 8, 2021 • 7 tweets • 2 min read
At the risk of getting involved in a discussion I really don't want to be involved in:

Excepting extreme circumstances, even very effective or very damaging policies won't produce discernable "spikes" or "cliffs" in COVID-19 outcomes over time.

That includes school policies. "There was no spike after schools opened" doesn't mean that school opening didn't cause (ultimately) large increases in COVID cases.

Similarly "There was no cliff after schools closed" doesn't really mean that the school closure didn't substantially slow spread.
Mar 7, 2021 • 5 tweets • 1 min read
I am in this picture, and honestly it's the most important thing. Poorly designed quality assessment metrics are arguably even more objectionable than poorly designed primary methods.

Two examples of VERY poorly designed quality metrics are the Newcastle-Ottowa scale and SciScore.

Compare to something like ROBINS-I (which is really good).
Feb 16, 2021 • 4 tweets • 2 min read
YES!!!

Most days in metascience, it feels like the odds are impossible, it's hard to believe that we'll ever make any progress at all.

And then every so often, something great happens.

This is a big deal for the future of science and publication, and I am STOKED! Full disclosure: I contribute every so often to the NCRC team under the fantastic leadership of @KateGrabowski and many others, and have been a fan of both NCRC and eLife since they started (well before I started helping).
Feb 14, 2021 • 46 tweets • 10 min read
Folks: There are serious statistical, design, language, and ethics concerns with that vitamin-D RCT.

AT BEST, it's completely meaningless due to negligent statistical treatment and design, but there's more "questions"

Help us out: avoid sharing until the critics have had time. Seriously, we (people who are frankly pretty good at this) are having a very hard time figuring out what the hell is happening in that trial.

Please give us time to do our work.
Jan 25, 2021 • 30 tweets • 7 min read
"Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation (PEACHPIE): A systematic strength of methods review" is finally available as a pre-print!

doi.org/10.1101/2021.0…

THREAD! One of the most important questions for policy right now is knowing how well past COVID-19 policies reduced the spread and impact of SARS-CoV-2 and COVID-19.

Unfortunately, estimating the causal impact of specific policies is always hard(tm), and way harder for COVID-19.
Jan 23, 2021 • 5 tweets • 1 min read
Broken record here, but speaking as a scientist who deals primarily with strength/quality of statistical evidence, the crux for just about everything in science lies in philosophy.

Many, if not most statistical evidence failures come from ignoring it. You don't need to read the complete works of 10k dead white guys, but it's incredibly valuable to dive down the "what does this even mean" rabbit holes.

Can't promise it'll make you more productive, but it will almost certainly make you a better analyst.