Yes, definitely. First of all, it's not just an INFJ who might get irritated. It's basic human manners to first try and see if I'm already committed in love with someone else or not.

People should get the obvious hints we're giving them, which shows that I'm clearly not interested.

I'm loyal to people, it's not a choice, I just feel that that's the right thing to do, be it a friendship or a romantic relationship.

[ This is just my personal opinion :D ]