There’s a subtle but very real fallacy about backtesting that lots of smart quant-y people fall into. I’ve fallen into it many times. And arguably I still do, just in more and more subtle ways.
A thread 👉👉
1/n
So you have a trading strategy, and you want to backtest it to see if it’s any good. Being good boys and girls and others, we know we mustn’t overfit to the data we already have.
We know that historical data is precious gold, and it must be used carefully.
2/n
Well, imagine I propose the following solution: build a model of the market in all its gory detail: fat tails, heteroskedasticity, vol clustering, etc etc. I calibrate this model using historical data, and it’s pretty good.
It's awesome in fact.
3/n
Why did I do this? Well now run this market model with different random seeds and I get different histories. Each one drawn from the same distribution (or meta-distribution, if you’re fancy) as real history.
So I get to backtest my strategy on all those fake histories.
4/n
Yay! With more data, I get better statistics on my strat. EV, drawdown, etc etc. There is much rejoicing!
Or is there?
5/n
Sorry, this doesn’t work. Here’s why...
6/n
Let’s think like information theorists. We take the whole history of prices and trades and everything.
A massively long multivariate timeseries of stuff.
A “model” is a thing that takes that massive amount of data and reduces it into a more manageable set of parameters.
7/n
These parameters can then be used to spit out reconstructions of the original market data. It’s dimensionality reduction.
If your market model is good, you reduced the # of dimensions a lot, but only lost a tiny bit of information.
This is good.
8/n
But you still lost *some* information. And perhaps more pointedly, you can’t have manufactured any NEW info.
What will that backtest on those fake histories generated by this model tell you?
9/n
From an information theoretic point of view, they CAN’T tell you anything you couldn’t otherwise have gotten from the original data! 😢
10/n
AHA you object!
But what about my prior beliefs?
The structure of the market model encodes my priors about the world.
11/n
Yes it does! And that’s a super useful thing for it to do.
But SO DOES YOUR TRADING STRATEGY!
All you’ve done is traded one problem (backtesting your strat) for another (making sure your market model accords with your priors, and reality).
12/n
That’s usually not a win.
Anyway, I’d appreciate some feedback on this. I’m not sure I did a great job explaining the main point, to be honest.
13/n
PS. Don’t @ me with complaints that I’m dismissing resampling methods like bootstrapping. I’m not, that’s different. Perhaps a conversation for another day.
14/14 FIN
• • •
Missing some Tweet in this thread? You can try to
force a refresh
"In all honesty, most finfluencers have pretty shit takes on the market. It kind of arises based on the mismatch between the skills required to actually trade and the skills required to market oneself."
"I'm a smart engineering/CS/math student/graduate. How do I get into quant trading?"
A thread. 👉
1/n
2/ I'm going to preface this by asking an important question:
Why trading?
Almost every outsider has an idea of what trading is that's pretty far from the reality. It's not "deploy cool ML models on gigabytes of data". It's:
3/ - Do I have enough cash margin in account JP44315A?
- Why does the data format disseminated by broker X suck so much with this new update?
- How do I optimize my tax footprint?
When Elizabeth I died, noblemen had to make a huge show of her funeral in order to prep people for the accession of James I (James VI of Scotland).
With modern eyes, it’s cool to see how critical the “show” was to a government’s legitimacy. Government meaning the king, obv.
2/
This wasn’t some unnecessary extravagance. Shutting the city down for a day or two for the funeral procession was a critical part of establishing the legitimacy of the next king.
Oh and by the way, yes James II really was a horrendously bad king.
3/