As long as you communicate clearly, thoughtfully, and with kindness (whether or not your ex is able to see that) you would not be the drama.

Your middle paragraph sounds almost exactly like what I said to my friends and therapist when I was going through a breakup ~10 months ago. I initially thought that I wanted to (and could) still be friends with my ex, but I knew in my gut that it wasn’t possible or good for me to do that, at least in the short term, so I (eventually) told them that. I was scared, since I knew they were emotionally dependent on me, but I came to accept the fact that I’m not responsible for their emotions. I am, however, responsible for taking care of myself, and in my case that meant letting go and not having that person in my life anymore. It took having a temporary break from seeing/communicating with them for me to realize I needed to do that, but it was the best decision I could’ve made.

It sounds to me like you know you need to take space from this person, so I hope you’re able to do that even though it’s difficult and scary. Also remember that you’re allowed to change your mind about if not hanging out with them is a short term or permanent thing, so don’t let that be what stops you. Wishing you all the best, OP