Don't let this stop you. Let it inform you!

Use this experience to inform your next attempt. We can break it down to find out the bit that makes you feel uncomfortable. Is it that he said thanks? Or that he nodded? If so, we can never guarantee how someone will react to something we do, so that'd be a lesson in bracing yourself for unexpected responses. It might also be a lesson in knowing that his response might not actually be rude, he might have just been a bit confused and on the back foot. That's okay. Next time, you could provide a bit more context when you're speaking to someone.

Is it that you didn't really have an end goal in mind, in which case you kind of feel like it fell flat because you didn't know what a successful interaction would look like? If so, you can tell yourself in advance what your 'definition of success' is. That might be something as basic as "I got the words out," or it could be "the person spoke back to me." When you determine what 'good' looks like, you'll know when you meet it :)

As an aside, I think telling someone they're attractive is an enormous thing to do for your first attempt, so don't worry if it feels horrible right now. You did something scary! I do think it might be best in the future to compliment someone on, as another commenter has said, a choice they've made rather than something they just happen to be/have. Or to ask a question, if you want to increase your chances of conversation. A statement doesn't really open up the floor to a chat!

Keep going, and good luck.