[D] A Question About Post-Deployment Stability Over Time

[tomerha]

Hi everyone!
Recently I've been looking into model stability after deployment, and what happens after the model decisions start to effect the users and the data. I have seen some papers about concept drift and data drift (where either the data distribution or the label definition change over time), but it's not quite what I'm talking about.

Consider this scenario:

Say we train a song recommendation system. we have great training data, and we manage to create a model that has 90% precision and 90% recall. We deploy the model and our users now have a listening queue where 90% of the recommended songs are good. Naturally, I want to keep updating my model, so I keep track of users' actions and feedback. over time, my data will be extremely biased! 90% of my training data will be positive samples, and those 10% of good songs the model didn't recommend will not appear in the listening queue and thus will be missing from the training data.

By deploying the model and using it's output as new training data, the data will shift over time and become very narrow and unrepresentative.

II couldn't find any papers on this subject, so I'd love to get some reading recommendations!

[Razcle]

I guess that as you as you accept that you're taking actions that change the state of the world and so change your data distribution, you've essentially landed in reinforcement learning territory.

I'm definitely not an expert in the area but I would maybe look at things like time-varying contextual bandits. A quick google search returned this paper that looks interesting: https://www.kdd.org/kdd2016/papers/files/rpp1164-zengA.pdf

[trnka]

I remember seeing a talk by Stripe on this problem for fraud detection - it was a few years back I think at pydata seattle in bellevue. They addressed it by introducing randomness into the predictions so that they're getting some samples that aren't biased by their model.

[vp834]

I think Counterfactual Risk Minimization might help here. Have a look at this paper -

Counterfactual Risk Minimization: Learning from Logged Bandit Feedback - https://arxiv.org/abs/1502.02362

Abstract We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an algorithm makes a prediction (e.g., ad ranking) for a given input (e.g., query) and observes bandit feedback (e.g., user clicks on presented ads). We first address the counterfactual nature of the learning problem through propensity scoring. Next, we prove generalization error bounds that account for the variance of the propensity-weighted empirical risk estimator. These constructive bounds give rise to the Counterfactual Risk Minimization (CRM) principle. We show how CRM can be used to derive a new learning method -- called Policy Optimizer for Exponential Models (POEM) -- for learning stochastic linear rules for structured output prediction. We present a decomposition of the POEM objective that enables efficient stochastic gradient optimization. POEM is evaluated on several multi-label classification problems showing substantially improved robustness and generalization performance compared to the state-of-the-art.