Thread by @kdpsinghlab

I had the opportunity to work with @AndrewLBeam and @bnallamo on an editorial sharing views on the reporting of ML models in this month’s @CircOutcomes. First, read the guidelines by Stevens et al. Our editorial addresses what they can and can’t fix.

ahajournals.org/doi/10.1161/CI…

First, everyone (including Stevens et al.) acknowledges that larger efforts to address this problem are underway, including the @TRIPODStatement’s TRIPOD-ML and CONSORT-AI for trials.

The elephant in the room is: What is clinical ML and what is not? (an age old debate)

What’s in a name clearly matters. Calling something “statistics” vs “ML” affects how the work is viewed (esp by clinical readers) and (unfairly) influences editorial decisions.

How can readers focus on the methods when they can’t assign a clear taxonomy to the methods?

Even the use of frequentist statistics don’t make a method “stats.” Conditional inference trees are an ML method where p-value is a tuning parameter!

Our proposal? Don’t call things “ML” in title. Just call them what they are, eg lasso regression, or don’t call them anything.

Wait, what? Don’t call them anything in the title?!

Think about it.

How many association studies do you read without the word “logistic regression” in the title? Tons!

So you can say “prediction model” in the title and name the method in the Methods.

But if the method is part of the innovation, name it.

Calling a penalized regression model an “ML model” will draw the ire of statistical readers. Calling a single “decision tree” anything but a decision tree will draw the ire of everyone!

Call it what it is.

A natural question that Stevens et al recommend addressing is why chose the approach you did. It’s always good to provide a reason...

BUT:

Your rationale will basically need to boil down to your expectation of signal-to-noise ratio and data quality/volume. Tread carefully.

Adhering to reporting guidelines is extremely important.

Why?

It’s the only way to catch bad science. The problem with bad science in prediction modeling is that many of the common errors inflate apparent performance.

Mo’ performance = mo’ higher impact journal = mo’ problems.

Recently, @DrLukeOR put together a flamethrower of a thread arguing against the sharing of models. I disagree. While sharing models shouldn’t be a prerequisite to publication, the current state is pretty pathetic. Very few ML models shared.

There are LOTS of ways to share! 👇

Recently a collaborator at another institution sent us a set of pretrained models in #rstats .rds format. It was a beautiful moment. It almost brought a tear to my eye.

Those types of moments should be common, not rare!

So, in sum, we advocate for:
- Clear reporting
- Calling methods by name
- Sharing models and not just code
- Making sure ppl w/ appropriate methods expertise participate in editorial decisions

In other words: doing ML in a way that can maximally help patients.