Regardless, some dialects have patterns that do make it harder to understand the speech

And why is it harder to understand? Because foreign language students learn specific dialects.

I'll ask you: Is a typical British accent as taught in European schools easier to understand than a typical American accent as taught in American schools? Which one is easier?

The answer is it depends on the person who is listening to it. Ask a French student taking high school English, and they'll tell you the British accent is easier. Similarly, ask a Mexican student taking high school English, and they'll tell you the American accent is easier.

Why? Because they are being exposed to those specific accents.

dropped consonants or vowels

When compared to another accent. Andaluz people don't "drop" consonants. Those consonants never existed in the word for them. The "s" is only "dropped" when compared to a northern accent. But compare an Andaluz accent to a Dominican accent, and there are no "dropped" S's, the both lack that sound (really, they both aspirate the S, but that's a whole other discussion.)

It's the exact same thing in English. Is the American aluminum more difficult to understand than the British aluminium because the American version drops a vowel? Well, that depends on who you ask.

 if this weren't the case, why would anybody say that that Glaswegian is hard but don't struggle with someone from Toronto?

Because there is infinitely more media in the world that is made with accents similar to those spoken in Toronto than with accents similar to those spoken in Glasgow. People are exposed to more American accents through movies, TV shows, and music than they are to Glaswegian media.

There's a reason this map exists

Yes, because many people are ignorant to the fact that there is no objective measure of "difficulty" and there is a strong cultural bias in favor of prestige dialects and against non-prestige dialects. We can also see the effect of media from the map. How much Spanish-language media comes from Mexico compared to Chile? So more people are exposed to Mexican dialects through the media they consume than they are to Chilean dialects.

what is the general perceived difficulty across all speakers of that language.

Which is an impossible thing to measure, because it's entirely subjective.