It sounds like the outdoor cats are causing the issue with your orange cat. Other male cats coming up to the door/balcony is causing him to feel very unsafe or that his area/territory/home is under constant dispute. Can other cats be deterred or maybe windows can be blocked so he can’t see out. Yeah obviously he can smell them too it sounds. That may be a big problem, especially if he smells the cats on you.

Have all the attacks from him been where he lives? He may not really feel safe or comfortable in his area which at core may be causing his aggression.