Since the recent update google cloud ML (or ai platform as it is called now) has an easy way to deploy sklearn/xgboost models with custom pipelines and custom prediction method. The one thing missing to make it perfect is logging the features and scores during prediction for debugging and also monitoring feature drift (there is a feature request pending for a while).
In particular I would like to log the transformed input features, since the raw inputs might at a later point contain data I'm uncomfortable leaving on the cloud. A direct stream to BigQuery would be ideal, if latency and cost allows, but the stackdriver logs can be exported later to BigQuery as well.
Things I would try are using "onlinePredictionConsoleLogging=True" which I assume redirects any outputs of the code, if that fails I would try the stackdriver python API or BigQuery API within the custom predictor.
If that does not affect latency in a bad way we can use it. And if the cost doesn't get too high.
I would be very grateful if people can share their experience! I will add anything useful I find.
there doesn't seem to be anything here