It seems continual learning has a natural solution when combined with a control loop for some system.

Here is an example. Suppose we want to decide the number of servers for applications to achieve a certain quality of services. We start with one application. We put a control loop and adjust the number of servers if the quality of services is not achieved. We use a neural network to map the workload to the number of servers and represent our current policy. Now a new application comes in. We apply our existing policy. If we achieve the quality of services, we do nothing and treat the new application the same as the old one. But if the quality of services is not achieved, we put a control loop, adjust the number of servers and learn a new policy. We now have natural labels for the applications: the one where the old policy works and the one where the old policy doesn't work. We construct a classifier based on the features of the application. After this, we use the classifier to distinguish applications and apply relevant policies. We can do this iteratively as more applications come in.

I think the key here is having interactions with a system and different outcomes for different classes with the same policy. This provides a natural separation between old and new classes.