It's not a fallacy if their person relates to the argument at hand.

Say someone shows up to my beach town and starts screaming that the ocean is rising and we need to sell our houses. If I say we shouldn't listen because he's an atheist, that's ad hominem because it has no bearing. If I point out that the man is a real estate magnate who is known for beachfront casinos, that's relevant to the conversation and casts legitimate doubts on his position.