,
9 tweets,
4 min read
Read on Twitter
In last week’s tweetorial* about the area under the ROC curve, I wrote that “there are some exceptions when I would not rely on AUC, but these are rare.”
One of these exceptions is the Gail model for the prediction of breast cancer.
* see here:
One of these exceptions is the Gail model for the prediction of breast cancer.
* see here:
When I tell people that risk models with low AUC are unlikely useful, I generally get the Gail model as a counter-example. The AUC of the Gail model is mostly <0.60, which is low and unlikely useful by my standards.
But the Gail model has something most risk models don’t have.
But the Gail model has something most risk models don’t have.
The Gail model has a relatively rare risk factor that has a strong impact on disease risk: family history. Having 1 or 2 first degree relatives with breast cancers increases the risk of disease by several-fold. See all effect sizes (RR) in the table.
Rare risk factors increase the risk for a small group of people but do little for the rest. The risk distribution looks like a ‘normal’ distribution with a long flat high-end tail—if the study is large enough to have enough people with the rare risk factor.
Risk distributions often have high-end tails but when they are less flat, they come with higher AUC.
AUC is a global metric, not insensitive but a measure for the big picture. AUC doesn’t easily show the impact of rare risk factors with strong effects. You may see a steeper ROC curve in the corner of the plot that represents the high-end tail of the distribution.
The Gail model does not always have a low AUC. Here is an exception where its AUC was 0.74. This study was conducted among women attending a Family History Clinic and all had a family history of breast cancer. The rare risk factor was common in this population.
Don't use the Gail model as an example of a great model. It isn't. At their threshold, the true positive rate (0.46) is slightly higher than the false positive rate (0.34). The model identifies women with a family or biopsy history, but does it do more than that? I don't know.
More on this can be found course manual on prediction research, including many references to key methodological papers (update is on its way). The slides of this thread can be downloaded here: cecilejanssens.org/slides-from-tw… … (later today)
