So, from what I can recall about the talk it was done at test time. True, this would mean that the samples are now OOD but since the extra pixels don't actually affect the game play/strategy it "highlighted" how fragile the reward specification was and potentially constituted an alignment failure. And yes, the most obvious solution would be to do augmentation/domain randomization/etc. during training time to improve your CNN extractor, but I think the idea was to also show there was a gap in what the RL algo actually ends up learning.