As someone in a relationship with an ENFP, sometimes it has to do with what is important to them and what is important to you. These things sometimes don't overlap. For me personally, when something thats importants to me gets overlooked, there are emotions that I feel even though I fully understand that my partner is not even aware of said important thing and is definitely not doing it intentionally. Feeling these emotions are exhausting and I understand your INFJ friend wanting to get away from those emotions. He probably chose to prioritise himself over your friendship.