Thread by @TaylorPearsonMe

This chart from 538 actually shows one of the main problems with many types of predictive models.

Your greatest volatility always exists in the future so any purely historical analysis is clipping off the tails.

To quote @paulg, history is merely all the data we have so far

https://twitter.com/FiveThirtyEight/status/1224167401287245824

Models with high BIAS tend to be simple and generalize well out of sample, but they may leave some explanatory power on the table

i.e. middling backtest but live results are more likely to resemble simulated results.

Models with high variance are more tightly coupled with the training data. They are highly explanatory in-sample but may not generalize well on unseen data (i.e. great backtest, poor live results).

Based on the assumptions you plug into the model, you can tradeoff between high bias and high variance.

I would argue that 538 (and many other people making predictive models based on past data) bias too much towards high variance.

The greatest vol always exists in the future so any purely historical analysis is clipping off the tails, creating blow up risk.

You have to build in margin of error, favoring higher bias

E.g. In investing, if you are trying to decide how much leverage to add to an investment portfolio, figure out the largest historical drawdown and double it (at least)

That will obviously reduce returns in the short term, but it increases the ergodicity (taylorpearson.me/ergodicity) of the portfolio, making it more robust and less susceptible to blow up risk.

This is bad for business at 538 because it would cause the model to show less volatility, making it less exciting to watch and generating fewer page views.