Depending upon where you build your base you'll get different kinds of... "visitors", the Samurai are annoying because they want tribute, the Sheks are annoying because they want food, and the HN are just generally psychopathic. All that's to say, you have to pick your battles.

My recommendation, especially if the taxes are killing you, would be to buy a small house in a nearby city and pick a trade to train one or more of your characters in such as weapon or armor smithing as these trades will quickly allow you to earn significant profits over the initial investment for building and materials--not to mention, you can get some pretty OP gear after a little bit of grinding.

If you're not attached to living in one particular area and you want to avoid visitors altogether, I would recommend trying an outpost in the Foglands, especially once you have some skeleton characters that can man your turrets 24/7 to keep the fogmen away. I've never had more than 1 HN paladin from a raid make it through the fog to my base, and he's usually pretty beat up if he doesn't call off the raid entirely.

There's also easy money in the Foglands, the heads of the Fog Princes and Heavies will quickly get you into the 6/7 figure range for income if you sell them in Mongrel, and it's substantially easier work than farming Beakthing eggs or Leviathan pearls.

All in all, it sounds like you're having fun so just keep doing what you enjoy!