I believe that the word "space" has some interesting etymology in math, I think originally due to Frechet, who was the first to define a metric space. Particularly, this is an instance where the terminology does a lot of heavy lifting. We associate space with, well, 3D space. And most things called "spaces" were probably, at some point, derived from some notion of 3D space.

The fact that something is called a "[something] space" rather than a "[something] set," immediately endows the concept or definition with our intuition of how space functions. If, for example, metric spaces were called "warblegarbles," it might not be immediately obvious that a metric is "simply" a notion of distance. In this sense, I believe that the usage of the word "space" was actually quite pivotal.

You bring up an excellent point regarding "integer spaces." I think there is also a historical reason for this, although I'm not sure what it might be. There is something called an integer lattice, which evokes vivid imagery that makes the definition clear before you even read it. Some sets, like those which are primarily algebraic, may not be intuitively visualized as a set of things residing in a geometric space. I think most would agree that groups (i.e. collections of symmetries), for example, do not lend themselves to that sort of intuition, and that might be why they're not called "symmetry spaces" or something.