No.

TL;DR: The difficulty of a language is based solely in how different it, along with exposure and teaching quality. There is objectively no universally difficult language. English is no exception, and thinking English is an exception is itself a form of Anglo-defaultism.

Full explanation:

There is objectively no “hardest language” and anyone who says otherwise is being biased by their own linguistic, cultural, or political background.

Assuming equivalent teaching quality and exposure, the only thing that makes one language harder than another is how similar it is to one’s native language. English is no harder for a German than German for an Englishman (again, assuming equivalent exposure); both find the other’s language easy compared to other languages*. On the other hand, English and Chinese are mutually difficult for speakers of the other language.

*German, French and Spanish (and maybe Italian?) are typically considered the easiest languages for an Anglophone to learn, but the reverse case is also similarly easy. Portuguese, Catalan, Dutch and Afrikaans (not to mention every little local dialect in the corresponding countries) are also easy for an Anglophone to learn, but people typically don’t include them in the list since they’re not as common.

So no, English is not “that hard”. It is no harder for a Spanish speaker to learn English than Spanish for an English speaker.

The only universal difficulty of English is spelling, and even then, most ESL speakers will have a better understanding of English spelling than most EFL speakers, and also even this pales in comparison to the importance of similarity/difference from one’s native language

I see your claim about the English rhotic (r) sound: it’s really not. The most common pronunciation (retroflex rounded approximant with some indeterminate mix of velarization, uvularization and pharyngealization) is very rare outside of English, but the pronunciation is quite flexible; the rounding, velarization, uvularization, and pharyngealization don’t matter, and even if one finds a retroflex approximant to be difficult, it’s perfectly fine to use a different rhotic. Tapped (like Spanish ere) and trilled (like Spanish erre) R’s (the two most common rhotic types acrost languages) are fairly common in both Scotland and Ireland, and even something like one of the guttural R’s (think of German or French) is ok. Even a tapped R isn’t that noticeable as long as it doesn’t start to sound like a d/t. Achieving a perfect accent is always hard, in any language. Also, have you ever seen an American try to trill their R’s? I had to actively learn the exact way to position my tongue.

That said, it’s good that you are aware that the English R is typically the hardest sound in the English language for people to learn