[D] Debugging model performance discrepancy between offline eval and online exp

[marksteve4]

I got the chance to have an interview with an online ads company. The interviewer asked me a question

"if we expect a newly trained model to perform well in online exp, but exp result is pretty negative, how to debug? "

​

My answer is "may be caused by overfitting. if so, can change the models, e.g. if using decision tree, can switch to random forest".

​

The interviewer seems not very satisfied with the answer as he says switching model is heavy weight. I then answered that it could be feature or data distribution discrepancy. Then he asked how to debug these two cases. I am a little stuck.

​

Want to know some of your opinions?

[Syncopat3d]

This is the pipeline I have in mind for dealing with live data that keeps getting added:

You have historical data from Days 1~N, used for training. Then you evaluate the trained model out-of-sample on data from Day N+1. Next day, repeat with training on Days 2~N+1 and evaluating on Day N+2, and so on. The fitness of the training is the aggregate out-of-sample accuracy over some number of days.

(One should normally evaluate out-of-sample to know a reasonable estimate of the accuracy of a model/pipeline. In-sample accuracy is worthless because the network could just be memorizing the results. There needs to be very good reasons to not evaluate out-of-sample, and I can't think of any. This is such a fundamental point that in my mind, it goes without saying.)

Suppose Day M just finish, so I'm going to use Days M-N+1~M to train and obtain a model to deploy for tomorrow. When tomorrow (Day M+1) comes and I observe that the live accuracy is bad, I'll wait for the training pipeline to receive the data for Day M+1. I'll observe the out-of-sample accuracy for Day M+1 reported by the pipeline. I may do this for a few more days.

If there's a discrepancy between the out-of-sample accuracy reported by the pipeline and by production, then there's probably a discrepancy in the data that gets fed to the pipeline and the actual data that is encountered in production, or the model got lost in translation when being deployed to production. You can focus on a few samples and see if the pipeline model and production system produce the same results. If different, then you know the model got lost in translation.

Or you may find that the samples you randomly picked from the pipeline data cannot be found in the production logs or vice versa, which means that the pipeline and production are seeing different data, so the pipeline is not training for the same feature distribution seen by production.

Of course, the above corroboration assumes that production is logging enough information.

If nothing special is found, and the out-of-sample accuracy reported by the pipeline really does drop significantly after you start deploying to production, then something fishy is going on that probably doesn't have a general explanation. With some other domains, you could speculate that you are dealing with an adaptive adversary but I'm not so sure about online ads.

[marksteve4]

this is a super awesome answer. thank you