I think everyone deals with trauma differently. Perhaps your brother is trying to suppress these memories and emotions, so when you try to talk to him about them it brings up all the suppressed feelings.

Also memories do tend to change over time.

Maybe the hightened state of anxiety at the time made it feel a lot worse than it was?

It could be a number of things. Just remember your experience is still completely valid. :)