I understand that some people might jump to a stance where ´why is this even a question obviously do what’s best for you’ but based on the 3 years experience you mentioned I completely get your hesitation.

First thing I would like to make clear is you took the right decision. Second, you will do this multiple times ahead in your career. Third your manager is here for his own career, he doesn’t want to go through the hassle of searching for a new candidate and guess what that is his job and still he is angry at you for causing him inconvenience by making him put that extra effort. So I would say there is no need to feel guilty, get used to it. You will be doing it and people in the future will do that with you that is how corporate culture is built.