Thread by @ltrd_

How to understand your prediction model and use it in real algorithmic trading? 🧵🧵🧵

Get a cup of tea or coffee and enjoy new knowledge.

#Trading #MachineLearning #ExplainableML #xAI #Shapley

In AlgoTrading it is really important to understand why your model and strategy work or does not work. Having that knowledge, you can adjust your features, build a better model or use this model in other cases.

I do not believe in putting some data creating for 1 day and doing Neural Networks in order to beat the market. In order to do it, you have to dig deep down into the data, and eventually you will find something good.

The models and data that I will provide here are 100% REAL. It is not random or synthetic data that will bring me 99% accuracy and beautiful plots. It is a real case with real data.

In HFT you have a really low signal-to-noise ratio so your model will not be the same as you have at university lectures.

Assume that you have your model that wants to predict something (in my case to predict 5 minutes log return). My model is XGBoost on 14 features [X1, …, X14].

In our case study, the model is not the most important. The most important is how to explain it and take more information from the fitted model. In order to do that we will use a python package called shap.

“Shapley values” is a concept from game theory that was brought after years into the Machine Learning models. Features are players, the outcome is a prediction. Our goal is to distribute the payout to the players (player importance)

Player importance in the game is an equivalent of feature importance in our model. The original papers and more can be found at this link → library.fa.ru/files/roth2.pdf

On the plot, you have all of the features. Each dot is an observation from a particular feature and its impact on model output. The more on the right, the more positive impact (in our case bigger log return within 5 minutes)

Red values correspond to higher values for the feature. Let’s consider feature X11. It is visible that most of the red values are on the right side. It means that a lot of high values of X11 corresponds to higher log return.

The most interesting features are those where the colors are split into two parts - one color for the left and one color for the right side. It means that there is a clearer impact of features on the outcome.

Those features in our case are X2, X3, X13, X11, X6, X14 and those are the features that you would like to scrutinize more. The most important thing is to be sure that it is intuitive that this feature impact prediction in this way.

For example, if you have a feature that is a measure for liquidity on the bids it should show you more red values on the right side because more liquidity on the bids means more buy pressure.

The second thing that you can do is to plot feature importance for a particular model - here it is XGBoost. You have this method implemented and you can use it after you fit your model.

You can see that the ranking is not the same as in the shap values and it is a really interesting problem in Explainable Machine Learning called the Disagreement Problem. See arxiv.org/abs/2202.01602.

One of the most valuable things when you look at the dependence plot for shapley values. You can see here the values of feature X14 and the shap value (impact on the outcome) with interaction with feature X1.

You can see linear dependence between 5 minutes log return and the X14 feature. And what is even more important is that X1 can split it into two parts. Low values of X1 correspond to lower values of X14 and higher prediction.

REALLY IMPORTANT: Every feature brings you some knowledge about the market and the model. Don’t screw it. Ask yourself what this feature or this interaction wants to tell you.

Maybe you should create two market regimes based on the interaction between those two features. Or maybe some feature has a lot of outliers and those outliers are matched with some values for other feature.

Why X13 has really strange shapley values for X13 > 0.0008 and why they are all corresponding to the really low values of the X14? These are the questions that you have to ask yourself.

This is how I do my analysis and how I learn more about the market. Remember that this is only my opinion and you should think for yourself. Nobody can do it for you.

It is really hard to monetize ML model in HFT. Even if you know that the price will be higher within 1 second there are tons of possibilities for how to use this knowledge, so you have to be creative.

I hope that it was an interesting analysis for you. Thank you for your reading. Good luck.