Thread by @MaartenvSmeden

Our new open-access paper is online at onlinelibrary.wiley.com/doi/pdf/10.100…

The non-technical story is this:
[thread]

Performance of a clinical prediction models in a setting where it is tested/validated is often worse than the setting in which it was derived. Suspects no 1 and 2: overfitting and differences in patient characteristics (“case-mix”) between derivation and validation settings
2/n

So we wondered: how about measurement error in the predictors?
3/n

“If it predicts, it predicts”

Some smart people have said: if a prediction model aims to predict an outcome based on predictors that are measured with error, the error itself is part of the prediction process and thus we don’t have to worry about the measurement error
4/n

But what if the way predictors are measured differently when the model gets tested?

Another way of thinking about that: what if the predictors are measured with a different measurement error in validation than for the derivation of the model?
5/n

A simple example: the predictor bodyweight, could be measured using a, let’s say, calibrated weighting scale or simply by self-report.
6/n

How about a validation study of the clinical prediction model using self-reported bodyweight when the prediction model was derived by calibrated scale bodyweight?

Spoiler: things can go wrong
7/n

We already knew discriminative performance of prediction models deteriorate with less precise measurements (e.g. self-reported weight instead of calibrated): jclinepi.com/article/S0895-…
8/n

Our new paper looked specifically at the impact on model calibration: the reliability of predicted risks.

This simple results with two quite highly correlated variables (~.70) isn’t pretty. This validation with less precise measurements makes model (look) overfit
9/n

Development with less precise measurements than at validation made the model (look) underfit
10/n

So less precise measurements at validation -> overfit, more precise measurement at validation -> underfit?

If only things were that easy…
11/n

For a variation in measurement strategies and precision between model development and validation we proposed the general term *measurement heterogeneity*.
12/n

The effect of measurement heterogeneity on validation performance can (to some extent) be predicted when perceived from a *measurement error point of view*.

Details in the paper: onlinelibrary.wiley.com/doi/pdf/10.100… …
13/n

So what? Well…

Apart from modeling and sample size issues (overfitting), poor(er) predictive performance at validation of a prediction model is often contributed to heterogeneity in ‘case-mix’ (characteristics of patients and setting). But…
14/n

… case-mix heterogeneity can relate to many things! Prevention of predictor measurement heterogeneity is a pragmatic solution to further optimize the external performance of a clinical prediction model.

Again, in addition to modeling strategies, sample size, etc.
15/n

What does this imply for clinical prediction research? Three implications:
16/n

Implication 1: ideally, prediction models are derived using predictor measurements that resemble measurement procedures in the intended setting of application. A mismatch is likely to result in miscalibration of the prediction model
17/n

Implication 2: one should bear in mind the implications of using a 'readily available dataset' for model derivation or validation as data/measurement quality directly affects estimates of predictive performance of the model
18/n

Implication 3: descriptions of measurement procedures at model derivation are essential for external validation of the model. Likewise, validation studies should describe the measurements procedures and possible measurement heterogeneity between derivation and validation
n/n