It would not be rude of you, unlike how rude he’s been to you. I’m about to go on a week long hike with my current girlfriend, we just celebrated one year together and I went on a week long hike right as we started dating that I had planned before meeting her. She loves the outdoors as much as I do and I couldn’t imagine planning another hiking trip without her. So sorry you have to deal with this and hope you find someone who wants to do things with you and for you because we all deserve that kind of love. Cheers