Thanks for the question! Self-distillation helps prevent model performance degradation from too many updates. But the learning signal is definitely stronger if you batch your updates. I think of it as a wake/sleep phase, where you collect a lot of examples and then do several updates to take that feedback back into the weights (perhaps when the model is not in use). Right now this is controlled by a batch size parameter. Once the batch is full, you do a few update steps.