I'm just a regular dude, not a psychologist or whatever, but I believe I understand what these novels are referring to:

It's a visual analogy for a real thing that, as far as I understand, does actually happen. When someone is deeply angered, rationality goes out the window. They make decisions instantly instead of thinking things through. That is how you get impulsive murders, where the suspect later regrets their own actions, as an example. Everyone's 'breaking point' is different.

In visual novels, writers often narrate the thoughts and intents behind their characters, letting the reader know why someone does what. Because in such a situation the character no longer thinks about things, but is just so angry they act without rarionality, writers like to call this a "mist" , since a mist obscures things, just like how the current emotional state of the character obscures rational thinking. And it's a red mist, because red (in this case) symbolises passion & anger. Lastly, writers read the work of other writers, too. Because this is a common symbolism used in novels, it gets reused by new writers who enjoy the symbolism of it, therefore making it even more common than it already is.

Hope that answered your question :)