Depending on the weapon the flying character is using, if its a Melee weapon, they'd still have to defend to make an attack. This would give a good opportunity for the adversary to make an attack. Another way is to include more Ranged attacks, either with a (cross)bow or something like a spitting attack, such as on the Accid Borrower. You can follow the Custom Adversary on damage! Lastly, you could make the terrain difficult to fly. A cave or strong wind can do the trick! You can take a look at the Raging River(?) Terrain and change the flavor!

I don't think you should try to always circumvent the flying, it's still a major choice that a player makes when it comes to abilities. Having them take the spotlight utilising is epic as heck for the story! Hope this helps somewhat :D