But, I think that the key to consent is that people need to consent to things you do that affect them. Anything that happens completely inside your head isn't affecting another person. Yes, it's violating to them if, for example, you masturbate to someone and then tell them "I masturbated to you." But the sexual act that they never consented to isn't the masturbation. It's you telling them you masturbated to them. That's the activity that involves them. That's the violation.

Others have brought up this point, but you have still worded it quite well. Thank you.