Thread by @AgustinLebron3

There’s a subtle but very real fallacy about backtesting that lots of smart quant-y people fall into. I’ve fallen into it many times. And arguably I still do, just in more and more subtle ways.

A thread 👉👉

1/n

So you have a trading strategy, and you want to backtest it to see if it’s any good. Being good boys and girls and others, we know we mustn’t overfit to the data we already have.

We know that historical data is precious gold, and it must be used carefully.

2/n

Well, imagine I propose the following solution: build a model of the market in all its gory detail: fat tails, heteroskedasticity, vol clustering, etc etc. I calibrate this model using historical data, and it’s pretty good.

It's awesome in fact.

3/n

Why did I do this? Well now run this market model with different random seeds and I get different histories. Each one drawn from the same distribution (or meta-distribution, if you’re fancy) as real history.

So I get to backtest my strategy on all those fake histories.

4/n

Yay! With more data, I get better statistics on my strat. EV, drawdown, etc etc. There is much rejoicing!

Or is there?

5/n

Sorry, this doesn’t work. Here’s why...

6/n

Let’s think like information theorists. We take the whole history of prices and trades and everything.

A massively long multivariate timeseries of stuff.

A “model” is a thing that takes that massive amount of data and reduces it into a more manageable set of parameters.

7/n

These parameters can then be used to spit out reconstructions of the original market data. It’s dimensionality reduction.

If your market model is good, you reduced the # of dimensions a lot, but only lost a tiny bit of information.

This is good.

8/n

But you still lost *some* information. And perhaps more pointedly, you can’t have manufactured any NEW info.

What will that backtest on those fake histories generated by this model tell you?

9/n

From an information theoretic point of view, they CAN’T tell you anything you couldn’t otherwise have gotten from the original data! 😢

10/n

AHA you object!

But what about my prior beliefs?

The structure of the market model encodes my priors about the world.

11/n

Yes it does! And that’s a super useful thing for it to do.

But SO DOES YOUR TRADING STRATEGY!

All you’ve done is traded one problem (backtesting your strat) for another (making sure your market model accords with your priors, and reality).

12/n

That’s usually not a win.

Anyway, I’d appreciate some feedback on this. I’m not sure I did a great job explaining the main point, to be honest.

13/n

PS. Don’t @ me with complaints that I’m dismissing resampling methods like bootstrapping. I’m not, that’s different. Perhaps a conversation for another day.

14/14 FIN