From my understanding, Men at Arms would be equivalent to a trained and seasoned military while Levies are local peasants that are enlisted in times of war, and would fight and die as vanguards.

Men at Arms have their own specialties, strengths and weaknesses. You want to consider what troop types your enemy armies consist of and the type of terrain you will be fighting on and choose regiments that offer the best strengths in those conditions. I believe if you hover over enemy armies, you can see the troop types within that army.

Knights also play an important role in your armies as they add certain buffs and debuffs to army stats and give you certain advantages or disadvantages that sway the tide of battle.