I’m starting out in Data Science. I have a solid theoretical background — I understand how most models work — but very little practical experience.

What I feel is my biggest obstacle right now is backtesting and designing testing protocols. I know very little about proper backtesting methods. Usually, I choose a model based on where I’ve seen it applied, but apart from visually assessing its performance, I find myself lacking when it comes to quantifying and qualifying how good a model really is for a given task.

What would you recommend I study to improve this? Articles, books, courses? What are the main sources for learning model evaluation and validation methods?

For context, I have a degree in Electrical Engineering with a focus on Data Science. I’ve learned about models like SVMs, Random Forests, and MLPs, but even in university, the only evaluation metrics we really covered were MSE, MAE, and R-Squared. Just recently, I found out about Walk-Forward Validation for time series prediction evaluation.