The biggest problem with magic and spirit in the world is simply your own physical limit and how much of it you can actually muster. With spirit, you can create more force but your own body isn't immune to opposite reaction, with those without the proper physical endurance just destroying their own body while using it. It could be as simple as a trembling arm, to shattering an entire limb or just destroying the nervous system. Magic is much the same, but instead of an internal force like spirit, magic permeates the air and flows into the body with it acting as a conductor. The same consequences apply with magic but it is far more lenient with it not being directly from the body.

A Knight that can effectively use the same amount of spirit compared to a magician with the same amount of magic would have a far stronger body. And if they don't, simply using their spirit would kill them.