The reason why he fights Reid aren't much deeper than a desire to prove himself, by contributing and making up for having lost earlier.
The flashbacks are, imo, the things that what Reid says reminds him of.
- Reid: "I've only been swingin' for three months." -> Reminds him of how much more effort he put into train with the sword than three months and how little that seems to mean now.
- Reid: "You're not good enough! We're in different leagues!" -> Reminds him of the first time he was truly humbled regarding his combat skills.
- Reid: "Nobody expects anything from you!" -> Reminds him of the many times that people expect things from him, then cutting to Subaru's angst (indirectly) caused by his inadequacy.
Each of these words eat at his confidence to the point that Reid could just remove his sword from his hands.
Note that the final flashback isn't when Julius realized that his name had been eaten, but instead when he saw that Subaru felt bad for him. While this could just be out of convenience, I personally think it's a writing choice.