Thread by @HangukQuant

1\ While I have criticized many facets of applying AI in trading, the appeals fall on deaf ears. If you are going to use them anyway, let’s do it correctly.
A deeper look at one of the most common pitfalls of “AI quants”.

THREAD; The Curse of Dimensionality.

2\ In an earlier thread, I argued the boon of having synthetic data where datasets were sparse, for example in the crypto space. This technique is particularly important when we consider applications of statistical learning algorithms.

3\ The favourite garbage AI model is one when people just spam 10 different features into a neural network to get some price signals. Lets argue why this often fails.

4\ Let’s first familiarize ourselves with a terminology: the hyperspace. When we consider a model parameterized by a single value, that value can exist on a line (univariate). Bivariate models are parameterized along a plane, and tri-variate models can be defined on a space.

5\ Now, when we extend this to higher (n) dimensions on real values, we obtain an n-dimensional “Euclidean space”. In search problem settings, we often call this the “hyperspace”.

6\ So, these 10 features you selected for your AI algorithm exists as a point in the hyperspace, and your algorithm searches this hyperspace for potential alpha. Now that we understand our search space and objective, we can discuss the curse.

7\ Let’s pretend that our features are independent, and identically sampled. Let’s also assume that our AI model performs predictions based on local regions of training datasets.

8\ To illustrate, let’s assume that our algorithm picks references from points such that the training input is selected if it’s point is within 10% range of the test input along each dimension.

9\ Case: 1 Feature. When we have a single dimension, 10% of the possible training datasets are used for an arbitrary test input. If we have 1000 training data and we want to make a prediction, the predictions are made by 100 other training samples.

10\ Case: 2 Features. When we have 2 dimensional features, 1% of the possible training datasets are used for an arbitrary test input. This is because only 10% of the training data are nearby our first sampling axis, and 10% of training data are nearby the second sampling axis.

11\ If we have 1000 training data and we want to make a prediction, the predictions are made by 10 other training samples. Our prediction is has significantly more noise. For 3 features, our prediction is only dependent on just one training input.

12\ As the feature space increases, our learning algorithms requires significantly more training data to have useful results. There are different algorithms that respond differently to such dimensionality issues.

13\ Along with choosing the right algorithm, dimensionality reduction techniques, feature selection, eigensystem analysis and synthetic data are just some cures to this apparent curse that is the demise of those who insist on intelligent trading agents.

FIN