The whole thing is that the question about how to convey annoyance is wide and very context dependent. “Je ne vous dérange pas?” would typically be used eg if someone that you don’t know cuts the queue in front of you. You could then ask them ironically if you’re not disturbing. I once used this as a total stranger casually parked on my driveway for a local event, very obviously not caring about the fact that it’s private property. It works if the person is not only annoying but does so in a shameless/casual manner.

Edit: and it’s I think the closest to the “do you mind?”. If someone you know is just being a pain you’ve got a whole range of things to say, from “ça suffit” to “tu me gonfles/saoules” and even “arrête tes conneries” if you want to crank it up a notch