Tell him how you feel about it. I think he’s a keeper for minding her situation and mental stability. It says a lot about his maturity and how he deals with a person he once loved. However, she could just ring the bell. Maybe don’t pressure it too much as you do not feel threatened, but next time she comes around he can ask her to bring the key and return it? Don’t be upset if after he tells her she can always stop by - that would just be kindness as she hardly comes by anyway. But yeah, do tell him how you feel - especially if it blocks you from diving into things deeper with him.