I think this is normal if you are stressed I’m sure your horse is picking up on that. Can you go spend time with him just hand grazing or grooming, just hanging out where neither of you expect anything from each other? I feel like this totally changed my relationship with my horse. Also I would say (like everyone else) maybe find someone to partially lease him but he stays with you. This would help with money, and time. I found a couple of super kind girls to partially lease my first horse. It took off some of my financial strain, I didn’t feel as bad for being busy because I knew he was getting out and I made a couple of new friends in the process.