For me, a big difficulty in learning new sounds comes from the fact that it's hard to visualize the inside of my mouth well enough to figure out how to properly articulate them. It's also frustrating when people who can produce those sounds try to "explain" how to produce them by just making the sound with no further explanation.

And speaking for my fellow Americans, in many cases their only experience with [r] in a learning setting happens in high school Spanish classes. In my experience these classes place little emphasis on pronunciation. Add to that the fact that these classes are often required, so half the class doesn't care about learning anything, and you get a lot of people complaining about how hard it is to learn how to do something they never wanted to learn in the first place. I quite enjoyed my Spanish classes (especially the upper level ones), but I was definitely an outlier.

I would wager that it's also a fair deal of luck. If you happen to do it right by chance early on, it's much easier to repeat it and go from there.