Thread by @bennpeifert

"but my machine learning model can forecast asset prices!"

stationarity is one of the most important concepts in probability and statistics

the essence of its meaning is that the specific pattern you are trying to understand is constant in a probabilistic sense

technically, its unconditional joint probability distribution does not shift over time.

in blackjack, the rules of the game are known and constant, and based on the cards seen so far we can know the probabilities of the outcomes of the next hand.

in finance, the underlying structure of the world is complex, unknown, and changes over time. there is little in the way of objective, certain truth. any historical data analysis assuming otherwise under-states the uncertainty around forward looking forecasts.

textbooks focus on simple and obvious cases, like the fact that the levels of asset prices (as opposed to changes) are nonstationary. this is of course true and it is why twitter charlatans are constantly posting spurious-correlation graphs of levels of two variables over time

but much less academic emphasis is given to unpredictable changes in data generating processes for returns, volatility, correlation, etc.

oil prices going sharply negative in march of 2020 because of lack of storage capacity is one very simple example

academics do like regime switching models, but in practice these are much better at fitting historical data than understanding what the true "regimes" are and helping observe and predict change in real time

when one tries to fit a highly nonlinear model with many implicit or explicit parameters to a complex dynamic system with a data generating process that changes over time, the result is a great looking fit in sample and utterly useless forecasting going forward

most AI/ML techniques were designed for stationary problems with high signal to noise ratios - eg image processing . there are analogous tasks in finance they can be great for; automating manual tasks like mapping identifiers of unknown format comes to mind

but naive prediction of asset returns is not one of them. the tricks they teach in class (split-sample validation, etc) help on the margin but do not address the underlying issue that there is very little signal relative to noise, and signals shift faster than you can learn

this was always fun

https://twitter.com/bennpeifert/status/1370587104430845957

there are some very specific cases where this criticism is less true - high frequency market making for example where there is a massive amount of data over very short time periods where the underlying dynamics are reasonably constant and models can adapt quickly

and don't come at me with ADF tests, those can reasonably answer the question "is it completely crazy to run this regression or look at this chart" but not the rest

we can teach a computer to play Go because the game's structure doesnt change... so we can simulate a million games and train a neural network and the next game is exactly like the first million we trained on

again this is not about "machine learning is not useful", much the contrary. but you have to think about what kind of problems it is useful for, how you apply it in a meaningful way, and how to quickly recognize bullshit that is being pitched to you

also, I cast this thread in terms of finance, but the same story is generally true for other complex phenomena where the structure of the problem is not known or constant

e.g. "we're solving health care by using machine learning algorithms on disease diagnoses" no you're not

https://twitter.com/Pups_rainbowzzz/status/1587561713968336896?s=20&t=-pngUSHnWt7A4AYzhG46lQ