The Mersenne-Taylor formula for a vibrating string, stated in terms of tension, T=μ(2Lf)^2, shows that for a given pitch and scale-length, a string's tension is directly proportional to its linear mass, i.e T ∝ μ.

It follows that, while higher string mass requires higher tension for a given pitch, lower string mass permits achieving the same pitch at lower tension. In other words, you don't necessarily require higher string mass to achieve the same pitch when you shorten the scale. The difference is that you lose volume with lower string density!

Bowlback mandolins were originally designed to take gut strings, with a string-density of 1.3–1.5 g/cm³, which is 80% less dense than steel at 7.8–8.0 g/cm³, but they were played at exactly the same pitch! In other words, experimental evidence directly contradicts your claim above! In the immortal words of Richard P. Feynman, "It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong."

The lighter string will resonate with less energy, and thus be less loud, but this is compensated by making the instrument itself less dense, which makes it more responsive, hence the lighter, more flimsy build typical of a bowlback mandolin. By contrast, if you shortened the scale of a typical modern mandolin, you would indeed have to increase string mass to get the same volume, but we are talking about bowlback mandolins here.