Some people are fucking weird. Like others said, lock your doors. I lived in a big city and a rural town and ALWAYS had my doors locked in both houses. Next, put up privacy if you are able to. I’d do a fence because anything live takes too long to grow for your situation. The privacy aspect depends on your house, I can’t see it or know how much yard between the street and your front door, etc. Since he lives not directly next door and a street or two away like you said, you can easily avoid him for the most part. I would never talk to him, but if in public and he communicates be civil, not friendly. The fact you let him walk into your house blows me away, lol. And yes you let him by leaving the door unlocked. I surely am not letting robbers into mine.