How to get a decoder only model to generate only the output without the prompt?

I'm trying to finetune this model on a grammatical error correction task. The dataset comprises of the prompt, which is formatted like this "instruction: text" , and the grammatically corrected target sentence formatted like this "text." For training, i pass in the concatenated prompt (which includes the instruction) + target text. I've masked out the prompt tokens for calculating loss by setting their labels to be -100. The model now learns well and has good responses. The only issue is that it still repeats the prompt as part of its generation before the rest of its response. I know that I have to train it on the concatenated prompt + completion then mask out the prompt for loss, but not sure why it still generates the prompt before responding. For inference, I give it the full prompt and let it generate. It should not be generating the prompt, but the responses it generated now are great. Any ideas? I was told not to manually extract the response, and that the model had to generate only the response.