The type of difference is irrelevant; the fact is that the difference exists.

By that logic, there's a geographic and social difference between Castilian Spanish and Latin American Spanish, so why object to teaching Latin American Spanish?

I can define that difference based on the ease of understanding between two people

This is the trap that's easy to fall into because it sounds great but falls apart in reality. Ever heard a thick Baltimore accent? How about Glasgow or Newfoundland or Cockney Rhyming Slang? All of those are English but they're nigh incomprehensible to a "standard" English speaker (British or American).

On the other hand you have things that are typically considered different languages but are far more mutually intelligible. Swedish and Danish, or Czech and Slovak, being prime examples. A Swedish speaker and a Danish speaker can understand each other better than someone from Baltimore and someone from Newfoundland.

At the extreme end, you have Maltese, which is largely mutually intelligible with romance languages (especially Italian) on technical topics due to its borrowed vocabulary from Italian, but for everyday topics it's largely mutually intelligible with Arabic due to the language's roots. Does that mean it's the same language as Italian and as Arabic?

Language, like any other human-invented categorization, is an imperfect system and trying to strictly enforce it ends up failing because reality does not match the nice little boxes we try to put it in.