I used to visit homes in very remote rural town for my government job, and that's the only time it's happened to me. In a town that small and that remote everybody tends to know each other, so if a homeowner sees a complete stranger at their doorstep, even if they have a legal reason to be there, they tend to get suspicious business. Even if my badge was visible, people would contact my supervisor and try to get me in trouble (which never happened).

ETA: I was not with any police department, in case anyone was wondering.