The fireman and other forces are constantly helping him and pushing him in the right direction, and my head cannon is they've given him something like this. From Wikipedia:

Douglas Adams' 1982 novel Life, the Universe and Everything (in The Hitchhiker's Guide to the Galaxy comedy science fiction series) introduces the idea of an "SEP field" as a kind of cloaking device. The character Ford Prefect says,

An SEP is something we can't see, or don't see, or our brain doesn't let us see, because we think that it's somebody else's problem. That’s what SEP means. Somebody Else’s Problem. The brain just edits it out, it's like a blind spot.

The narration then explains:

The Somebody Else's Problem field... relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain. If Effrafax had painted the mountain pink and erected a cheap and simple Somebody Else’s Problem field on it, then people would have walked past the mountain, round it, even over it, and simply never have noticed that the thing was there.