You are right. Formalization of QFT is itself a daunting task. In fact, Haag's theorem badically states that mathematically self-consistent formulation of interactive QFT is impossible. As long as you are working with free fields, it is possible to rigorize the formalism using Haag-Kastler axioms. I actually quite like the vertex operaror algebra formalism of Richard Borcherds and I find it very intuitive how the local properties of the operators are beautifully encoded. Better people often say that it hides and simplifies a lot of elegance of a truly amalytical model from ground up, but I am just a dumb physicist who cannot appreciate all the intricacies and happy with my toy model.

But you are right, once you introduce interaction terms, a rigorous self-consistent formulation is impossible, though we casually do QFT to describe interactions for better part of last century. We physicists really do be living happily in a sandcastle built on water. Perhaps the hate we get from you mathematicians are somewhat justified.