Thread by @fjnogales

A Backtesting Protocol in the Era of Machine Learning papers.ssrn.com/sol3/papers.cf…

A financial paper with excellent thoughts about the cautions that should be taken in quantitative methods. My comments below: 👇👇👇

What relevant authors!
Including a Nobel prize!

The paper provides a protocol for empirical research in finance
And the motivation for the protocol is excellent...

It includes a data-mined strategy that is long in stocks with tickers with an "s" in its third letter and go short in those stocks with a "u" in the third letter.

The Sharpe ratio is impressive, the performance is both economically and statistically significant, generating a 6% alpha per year!

But would you invest in such a (non-sense) strategy?

This data-mined strategy was discovered by brute force, not by machine learning. It was trained and tested with very limited data. So, without any economic foundations...

The problem is that in finance, with limited data, there is usually more noise than signal.

And with so many models tested and retested, it's not difficult to find false positives (discoveries). And once published, these promising models rarely work as well as in the backtest...

So, a protocol to avoid these false positives is needed. And the authors provide such a protocol to backtest investment strategies.

Summary: excellent paper with very interesting reflections and ways to proceed to develop new and reliable investment strategies.