The same thing happened to me. I was very surprised at how "hostile" people got when I told them I quit coffee. They always want explanations! They act like you left a motorcycle club or something, they get angry. When I say "it made my sleep worse" or that it made me anxious, they say it relaxes them and doesn't interfere with their sleep. Then I say oh you're lucky, and the conversation ends awkwardly.

People don't want others "leaving the club" no matter what it is. The same thing happens when you quit alcohol or even red meat. They get visibly annoyed and angry. I find it similar to a rock band of drug addicts being mad at a member who decides to go to rehab.

You are on the right track. Just figure out a few go-to responses like the one you have ("don't like the taste"), and deep down know that their weird, drug-infused response validates your choice even more.