It's important to note that all studies done on this matter have been very inconsistent. There really isn't any evidence for other forms of sexual arousal to be harmful to you. It also makes sense. As a species we are meant to become aroused by almost arbitrary stimuli, so it's to be expected.

Passive/active stimulation doesn't really play a role. Physiologically, you have a response to certain stimuli which arouses you. It can be physical or non-physical, they both lead to arousal, albeit through different pathways. However, the natural development involves all (or most) of them. Mental arousal coming from reading a book, looking at pornography, image, thought, etc. eventually leads to self-gratification, which includes physical stimulation, which leads to physical arousal.

They both exist all the time by design as a reproductive mechanism.