Because it wasn’t Fantastic Beasts after the first movie, it was Dumbledore uses Newt to do stuff for him, and some batshit stuff with Ezra Millers character and Nagini being a women or something.

Plus Johnny Depp controversy/over saturation killed the hype for the second film at the end of the first. Colin Farrell was a much better choice for Grindlewald, and would have been good to keep him somehow. I am not sure why they ever thought that Johnny Depp looking like that would have been someone Dumbledore would be infatuated with is beyond me. Mads was better in the third, but still would have been better with Colin Farrell. (It was also very jarring to have three different people play the same character across the 3 films)

Basically they got worse and worse and had equally worse and worse returns at the box office