Quant investing: What is overfitting?
An overfitted trading strategy is a model that has been trained on a specific dataset and has learned the noise in that dataset, rather than the underlying trend. This results in the model performing well on the training dataset, but poorly on new, unseen data. This is because the model has learned the noise in the training dataset, and is not able to generalize well to new data.
Overfitting can occur when a model is too complex for the amount of data that is available for training. For example, using too many indicators or inputs in a strategy can lead to overfitting. Additionally, overfitting can also occur when a model is trained on a small dataset, or when a model is trained on a dataset with a lot of noise.
How do you avoid overfitting?
1. Sample size
The law of large numbers states that as the number of trials in an experiment increases, the average of the results obtained from the trials will converge toward the expected value of the underlying probability distribution.
2. Strategy simplicity
The more complicated the strategy is, the higher the possibility is that it has been overfitted during the optimisation process.
3. In & out of sample testing
Another way to avoid overfitting is to use out-of-sample testing, which involves testing the model on a dataset that was not used for training. This can help to ensure that the model is not overfitting the training data and is able to generalize well to new data.
In summary, overfitting is a common problem in trading strategy, it occurs when a model has learned the noise in a specific dataset rather than the underlying trend. This results in the model performing well on the training dataset but poorly on new, unseen data. To avoid overfitting, it is important to use a sufficient amount of data for training, to keep the model as simple as possible, and to use cross-validation, regularization techniques, and out-of-sample testing.
To find out more about the specific quantitative strategies we implement, please call our office on 02 8668 4877.
5 topics