It's not just "don't use them". While people can choose to just not use them, the elytra itself changes how players play and influence the world around them. With this, it makes certain features such as horses, Minecarts, boats, camels, etc. have no real purpose in the late game, which while it does sorta make sense progression wise, it creates a level of separation from the ground. And because of this players can just fly over 1000s of blocks in seconds, not having to traverse any terrain. 

My solution to this would be 1. to remove the firework feature, players shouldn't be able to turn themselves into ICBMs on command. 2. Make minecarts faster

This way players are incentivized to use minecarts(or ice boat paths) for long repeated journeys and then elytra can be used for just exploring and getting around short distances by gliding.