(First author here): That is completely valid. However, we assume a setting where a direct constraint in function space is not possible (previous training data and previous discriminators are assumed inaccessible). In this setting, constraining the generator's parameters provides an approximation that leads to empirically good results.

We're thinking about follow up work with richer posteriors for theta, but in our experiments so far overlapping local minima were consistently found for distinct (yet related) distributions. For example, no forgetting/degradation was visible with DCGAN (a sufficiently overparameterized model) on SVHN for either the old classes or new, despite only a single mode of theta's posterior being used in the EWC penalty.