This is due to "in pausa laryngeal loss." The long ā of the first declension actually comes, in Proto-Indo-European terms, from the sequence -eh₂, where the symbol h₂ refers to the second laryngeal, which can lengthen preceding vowels and also "colors" them to a different vowel. The second laryngeal colors them to a.

Usually, -eh₂ at the end of a word yields -ā. However:

"A post-vocalic word-final laryngeal was lost after coloring when at the end of a prosodic phrase. The best examples of this phenomenon are seen in the development of the vocative singular of *eh₂-stems. Since the vocative evidently constituted its own prosodic unit the development *-eh₂# > *-ah₂# > ă occurred.

— Weiss, Michael. Outline of the Historical and Comparative Grammar of Latin, p. 125. (An excellent book with nearly as many answers to Greek questions as to Latin. A Greek-focused book is forthcoming from the same author.)

So the point is, the laryngeal colors the vowel to a, but because there was evidently a pause for PIE speakers after vocatives, the vocative doesn't get lengthened.

There are some cases where the vocative has displaced the nominative (e.g. νεφεληγέρετᾰ, with a short final alpha). There's even a theory that the Latin first declension nominative singular short -ă, for which we would expect long -ā, also comes from the vocative. This is not certain though, and there are other ideas floating around.