#Medical #AI researchers: badly performed/described cross-validation is the most common reason I recommend major revisions as a reviewer.
CV can be used to tune models and to estimate performance, but not on the same data. See this diagram for doing both.
h/t 4 pic @weina_jin
CV can be used to tune models and to estimate performance, but not on the same data. See this diagram for doing both.
h/t 4 pic @weina_jin
@weina_jin The weird thing about CV in AI is that you don't actually end up with a single model. You end up with k different models and sets of hyperparameters.
It allows an estimate of generalisation for a *group* of models, but that is still a step removed from a deployable system.
It allows an estimate of generalisation for a *group* of models, but that is still a step removed from a deployable system.
@weina_jin For a more detailed explanation, see the "Nested cross-validation for model assessment" section of: ncbi.nlm.nih.gov/pmc/articles/P…
and here is the blog post from @weina_jin that reminded me to tweet about this topic weina.me/nested-cross-v…
and here is the blog post from @weina_jin that reminded me to tweet about this topic weina.me/nested-cross-v…
@weina_jin To show how widespread this is, a colleague just did a systematic review on LOS prediction, and AFAIR *none* of the papers reported appropriate CV methods.
That matches my experience, I have *never* reviewed a paper which satisfied me with their CV description on first read.
That matches my experience, I have *never* reviewed a paper which satisfied me with their CV description on first read.
@weina_jin My own opinion (which does not inform my reviews) is that CV for both model selection and performance estimation is way more trouble than it is worth. I discourage my colleagues from using it.
It still shocks me that TRIPOD recommends cross-validation :p
It still shocks me that TRIPOD recommends cross-validation :p
