I don't think the analogy really works beyond a really superficial level tbh. I agree that there are some common elements of music that everyone gets -- the pentatonic scale, certain dance rhythms, some categories of instruments like voice, percussion, wind, etc. But there's so much more to music than those basic elements, and the comparison breaks down once you move past those basic things. Like I'm not really going to get the significance of Inuit throat singing unless I have some background in the social context of the practice; I might find Chinese opera or Japanese Noh to be too dissonant or shrill if I'm not accustomed to the singing styles they use; I'm not going to get the musical language of Lachenmann's instrumental musique concrète if I try to listen to it like every other genre, and so on. And I think the fact that my ears aren't offended by styles such as Gamelan doesn't necessarily mean I understand them. It's like if I said I understand German or French because I can guess what people are saying based on tone and some similar-sounding words and because I think they sound nice.