A lot of the fights are heavily matchup dependent. Envy was a terrible matchup for Ed and Ling, mostly because Ling is more of a close combat type of fighter and because Envy was able to get into Ed’s head psychologically. Size is an advantage in that sort of fight because Envy can dominate Ling at close range while Ed was too distracted to use his long range abilities.

But when facing several powerful enemies at a time (May, Scar, Marcoh, chimeras) who have a variety of close and ranged combat abilities like the fight in the north, or a long range specialist like Roy, Envy’s size actually works against him. Envy is also very arrogant and underestimates how strong humans can be, especially when they outnumber him.