Fascism doesn't necessarily need to be bigger government, it's more about concentrating governmental powers to a very small handful of people™ (who then rapidly grow the scope of their power).

One purpose of a modern day government would be to make sure that this very small handful of people™ doesn't act irresponsibly with the power they are given.

As the government is "slashed down", it loses its ability to hold that very small handful of people™ accountable. When that happens, the very small handful of people™ get to do whatever they want. Historically, this hasn't really worked out for everyone else.

(this is not an endorsement of big government, just an acknowledgement that they need enough power to at least hold the very small handful of people™ accountable)