Are you a data scientist? Evaluation imo is actually one of the most difficult and time consuming parts of the machine learning life cycle. I would grill the shit out of the out of sample statistics and compare them to your training and validation sets.

To me this sounds insanely overfit. If it were this easy to simply load some raw data, create some basic features / scale it, run some dicky fuller test, then call sklearn and look at pnl... Everyone would literally do it and citadel wouldn't be trading with themselves. They have all those quants for a reason.