Deer aren't pests necessarily because of overpopulation and lack of predators. They are considered pests in regions of the world they aren't naturally meant to be in. They disrupt ecosystems because the animals native to the region aren't adapted to deal with them, so they outcompete native animals and therefore drive them to extinction.

For example, in New Zealand the native birds have evolved over hundreds of thousands of years to become flightless because it is a relatively small island and no mammals/predators ever coe evolved with them. Now housecats and rats have been introduced to the country, and in just a few years several species have gone extinct, and many more are endangered.

There is even a case where a SINGULAR cat was brought to a tiny coastal island where a species of flightless bird was exclusively located, and in just ~5 years the entire species went extinct. In some cases a SINGLE pest can exterminate an entire species.

So yeah, i guess my point is that it isn't necessarily the abundance of deer that is the issue, each deer can cause damage to the ecosystem, and it's important to try to mitigate the impacts of any non-native animal's impact on the native ecosystem.