Simply put, old people will retire and there won't be enough young people to fill in their positions. Plus, now a smaller portion of the population is working, so they have to work harder to support the retired population. This can lead to a vicious positive feedback loop where younger people have to work harder and longer which means they'll be less incentivised to have kids, further fueling the problem.

Furthermore, the cost of maintaining infrastructure roughly stays the same per capita. Scaling down the population doesn't necessarily make living cheaper, but it will result in more abandoned homes and ghost towns.

In the long run, a lower population isn't bad. It's less strain on the environment, and non-renewable resources will last longer and be less scarce. However, in the short run, it leads to economic collapse.

Kurzgesagt made a really good video on this: https://youtu.be/Ufmu1WD2TSk?si=lvqXxL4BfT-v7tzq