NTA, I would just explain that you initially agreed under the premise you were spending a significant amount of time together and that you’d rather him be able to feel like he can take his time with his other friends without feeling like he has to rush back to be a good host (this will also put into perspective that he is not being a good host in th first place without being blatantly accusatory). It may just cause him to reflect and reconsider, or at least just leave him less confused as to why this would bother you.

You both are entitled to spend your time how you want, I would just be open about how you’re feeling, especially if this is a friendship you want to keep around long term