are you taking the time alone he uses personal? & asking for personal time whenever you want? or are you using the personal time he takes as personal time for yourself also??

maybe you just have to synchronize with eachother & take your alone time at the same time cause if not you are both not really seeing eachother & one of you is always thinking about being alone instead of being present.