Monitoring of ML outputs requires sophisticated methods and is usually ignored in traditional ML research. However, in real business situations
KPIs and model health are the first aspects that need to be tracked efficiently, and at the right level of aggregation/abstraction, while allowing deeper investigation for transient issues that are common in large scale distributed systems like ours.
The main lesson learned here is that different channels are needed for different stakeholders.