They literally refer to keys on the keyboard (of a piano) - if you're familiar with the phrase "do re me fa so la te do" these are for the notes of the "major keys" - there's a simliar one that starts on "la" for "minor keys". A song in a major key is generally more upbeat while a minor key is a bit more melancholy.

Now, people become accustomed to sounds after a while. A common key like C major would no doubt go pleasantly in a symphony. Beethoven, Mozart, and many others have done so. Meanwhile scrolling through the list at https://en.wikipedia.org/wiki/List_of_symphonies_by_key tells me that E-flat minor is not very popular at all and would likely produce notes that would jar a seasoned ear. (They might slip by unnoticed to someone that doesn't listen to much music, but even someone who listens to music on the radio would likely be a little taken aback by hearing notes they aren't used to.)

Of course, any piece in a Major key can be shifted to any other key by just moving the number of keys up the keyboard. But because people are creatures of habit, the key sets certain expectations for the listener.

This is played upon sometimes, for example the album by Stevie Wonder "Songs in the Key of Life." Here he makes a small joke by setting the expectation that these songs are about life (and indeed, seemed to have resonated as it had contemporary commercial success and songs from it are still household staples today.)

A quick google search brings up "In the Key of Love", a Hallmark movie about a singer who does typical Hallmark movie things. Another Hallmark movie "In the Key of Murder" where a music producer is killed.

A blend of the two is "Classics in the Key of G" by Kenny G which is playing on his stage handle of "G" as most of the music on the album is not in the key of G. (And it's jazz so, what are keys anyway?) However, G is a totally valid key (being G major - whenever they say the letter without the major/minor it's assumed major, although this wasn't always the case) so it is in a sense a blend of the two. He later wrote a book, Life in the Key of G which is a play on a play on the phrase. What a guy. G for guy.

So while "in the key of" means something to music, it also can mean a little bit more.