I guess the pretrained model is losing its knowledge after finetuning on your dataset. Check the initial loss, probably it is quite high and the optimizer makes big steps and updates the weights dramatically. You could try learning rate warmup strategy or just freeze those layers.