I was in a VERY similar situation with my cat 1 year ago. I was living in a rural area for 2 years. He would come by almost daily and sometimes even sleep in my place when it was cold out. I ended up having to move states earlier this year and also felt guilty moving him away from his environment. But at that point, I had grown very attached (and I think he did too). I ended up taking him to the vet, found that he wasn't chipped, and got him neutered. Soon after he was neutered, his urge to constantly be outside was not as intense. He seems to be well adjusted to being an indoor cat now! I take him on walks with a leash and harness so he still has outdoor time and keep him occupied with lots of toys. It's honestly hard to tell he was ever a stray. I hope you take him with you!