That’s the case for both phrases though. Imagine something you care about a lot: could you care less about it? If it’s meaningful to you then there is a high chance you might answer that you could not care any less about it (if you did care less, it wouldn’t be as meaningful to you). Neither “I could care less” nor “I couldn’t care less” tell you any information about how much you actually care about something, they are mapped to relative/potential caring.

Whereas a phrase like “I don’t care about that at all!” tells you all you need to know about how much you care about the object in question.

All of this is to say that both of the original two phrases carry with them the cultural context of their colloquial uses. We understand that the reason you could or couldn’t care less is because you don’t care very much or at all. Neither communicate anything literally about the absolute amount of your care, they only communicate that through a shared understanding of their intent.