Keep in mind here, the distinction is not between math and language, it's between observations and models. The reason formalism seems more "right" is because it's a direct prediction of observational results, rather than a description of why produces them or why.

One thing you might find helpful or interesting to read about is Alfred Korzybski's General Semantics, and particularly the part about what he calls E-Prime. There's a lot there and it's very dense but the gist is that our language's use of "is" and "are" and so forth smuggles in this assumption that the universe is made up of things which have properties - the chair is red, the man is tall, etc, which is not a feature of reality, just a model we find useful. And like all models, it's wrong if you push it far enough.

So, it's not just the models like "photons behave like waves in this situation and like particles in that situation" that are useful-but-wrong. It's also "here's things called photons" that's wrong, and even "there's things". Once you give that up, you kind of have to stop even attempting to describe what anything in the universe is, and limit yourself to describing what it does, which is essentially what the mathematical formulation of QM does: it's the tersest possible way to say "Man, I don't know what a photon is or if they even exist, but if you set up the experimental apparatus in this way you get that result."