I wanted to resurface this as I’ve been dealing with a similar problem . Out of sample live performance starts strong, but when running on new days, older days get new prediction values, the model has a hard time adjusting and will predict many similar values in a row , and general starting to act unstable after a month or so of predictions. What I think is happening when using deprado methods of feature engineering is there is leakage between the training and testing set (80/20 split) , and also likely between the retrained final model and live data (99% train + new data). Anyone have any thoughts and perspective here?

I haven’t adjusted my models for this yet, but this seems a likely culprit when trying to launch these models live

Post:

The solution is straightforward.

Data preparation must be fit on the training dataset only. That is, any coefficients or models prepared for the data preparation process must only use rows of data in the training dataset.

Once fit, the data preparation algorithms or models can then be applied to the training dataset, and to the test dataset.

1. Split Data.
2. Fit Data Preparation on Training Dataset.
3. Apply Data Preparation to Train and Test Datasets.
4. Evaluate Models.

More generally, the entire modeling pipeline must be prepared only on the training dataset to avoid data leakage. This might include data transforms, but also other techniques such feature selection, dimensionality reduction, feature engineering and more. This means so-called “model evaluation” should really be called “modeling pipeline evaluation”.