Thread by @mikeharrisNY

Get anything you like. Thread about forecasting.

What is forecasting? Forecasting is basically number crunching for the purpose of making decisions and developing data-driven strategies. There is a whole array of methods, tools and procedures.

But before we even start: forecasting is both art and science. At times, it may be 90% art and 10% science. Why? Because reality is underdetermined by data. This indeterminism is fundamental and beyond the scope of this thread. This is subject of grad courses in phi of science.

Not all methods and tools apply to all forecasts. It's important to understand that application is domain specific. Different tools are used for weather forecast than those used for inventory and sales forecast.

Speaking about weather forecasts: the results from models cannot change the weather. But in financial markets, usually forecasts are used for trading and investment decisions. Therefore, there is reflexivity when forecasts involve financial time series.

Reflexivity occurs when price series affect forecasts and in turn, forecasts affect time series. Each forecasting agent does not have the same effect: the influence of a random analyst having a blog is tiny compared to influence of an investment bank.

In fact, with financial price series we may not know in certain regimes, whether it was the forecasts that drove the prices or the prices drove the forecasts. As it turns out then, financial forecasting is more of alchemy than science. Social score has more weight than methods.

Then, consider sales forecasting. Suppose the forecast is for 1000 units for the year but only 600 are sold. Then the company announces a sale at high discount and all 1000 are sold. The forecaster may celebrate but the forecast was wrong.

Confidence intervals are very important but rarely offered and even if they are, they may make little sense due to large deviations from normality. But offering confidence intervals is important. In finance those intervals are noise due to lack of sufficient samples.

It's important to understand that forecasting models with low error are data-specific and models do not generalize well across different domains. During forecasting conferences, different models win under different domains. That should ring a bell.

Speaking about forecasting competitions, you can never know if the best models are due to selection bias or they are truly good. Given a large number of participants, some of the may come up with models that will perform well out-of-sample by chance alone. But...

rarely there is a follow up to determine how these models performed in forward samples. Why? because in 95% of the cases these models fail in forward samples and this is not good for the competition business models.

Creating interesting and successful forecasting models is far from only data science. It requires creativity and thinking out of the box. Unfortunately, most companies wont take the risks. Forecasting libraries have reduced forecasting to programming application.

Machine learning offer new ways of dealing with thousands of variable and creating abstract models. But abstraction is also a drawback. Nevertheless, ML offer more possibilities for extracting interesting futures from the data but bias-variance trade-off is an issue to deal with.

All in all, forecasting is no different from the old alchemy practice of trying to transmute metals to gold but in modern ways obscured by mathematical models and programming languages. Is it worth the time, effort and resources. Maybe, or maybe not. It depends.

Finally, my presentation about reflexivity in financial forecasting for M4 conference in Dec 2018 can be found here priceactionlab.com/Blog/2018/12/m…