Even for science and medical applications, I am becoming weary of fine statistical modeling efforts, and believe that we should standardize on a handful of powerful and robust methods.
An opinionated thread to give context for
1/8
An opinionated thread to give context for
1/8
First, analytic variability is a killer.
eg in "standard" analysis for brain mapping onlinelibrary.wiley.com/doi/full/10.10…, for machine learning in brain imaging
sciencedirect.com/science/articl…
or more generally in "hypothesis driven" statistical testing
go.gale.com/ps/anonymous?i…
2/8
eg in "standard" analysis for brain mapping onlinelibrary.wiley.com/doi/full/10.10…, for machine learning in brain imaging
sciencedirect.com/science/articl…
or more generally in "hypothesis driven" statistical testing
go.gale.com/ps/anonymous?i…
2/8
We need weakly-parametric models that can fit data as raw as possible, without relying on non-testable assumptions.
Machine learning provides these, and tree-based models need little data transformations.
3/8
Machine learning provides these, and tree-based models need little data transformations.
3/8
We need non-parametric model selection and testing, that do not break if the model is wrong.
Cross-validation and permutation importance provide these, once we have chosen input (endogenous) and output (exogenous) variables.
4/8
Cross-validation and permutation importance provide these, once we have chosen input (endogenous) and output (exogenous) variables.
4/8
If there are less than a thousand data points, all but the simple statistical question can and will be gamed (sometimes unconsciously), partly for lack of model selection. An example in neuroimaging biorxiv.org/content/10.110…
I no longer trust such endeavors, including mines.
5/8
I no longer trust such endeavors, including mines.
5/8
For thousands of data points and moderate dimensionality (99% of cases), gradient-boosted trees provide the necessary regression model
scikit-learn.org/stable/modules…
They are robust to data distribution and support missing values (even outside MAR settings arxiv.org/abs/1902.06931)
6/8
scikit-learn.org/stable/modules…
They are robust to data distribution and support missing values (even outside MAR settings arxiv.org/abs/1902.06931)
6/8
For thousands of data points and large dimensionality, linear models (ridge) are needed.
But applying them without thousands of data points (as I tried for many years) is hazardous. Get more data, change the question (eg analyze across cohorts).
7/8
But applying them without thousands of data points (as I tried for many years) is hazardous. Get more data, change the question (eg analyze across cohorts).
7/8
Most questions are not about "prediction". But machine learning is about estimating functions that approximate conditional expectations / probability. We need to get better at integrating it in our scientific inference pipelines.
For more, push me to write a paper on this.
8/8
For more, push me to write a paper on this.
8/8
