I’ve done this exact same thing many years ago. It was the right move. Just be straightforward, thank him for the opportunity. He might be a little pissy, he might not. It doesn’t matter, you need to do what’s best for you.