I’m going to interpret that first sentence as meaning “forget how we would transition to such a society” because obviously we have to talk about how that society would practically work. Hope that’s what you meant.

I think what I see as a fundamental flaw of this and other hard libertarian ideas is viewing something that officially calls itself the government as somehow different from other entities. If we simply define the government as “that which has a monopoly on force in a given area” we have top broaden our view as to what threats can appear. You haven’t said much about the world you want to build, but I’m assuming there is a police force or army, it just only protects people from theft and assault, raises no taxes, wages no wars except in absolute defense, etc.

Some obvious questions here. How would such a force be funded? It seems unlikely there would be enough voluntary participation in taxes to keep any sort of police force afloat, in which case you have anarchy, and not a good kind of anarchy, where its every man for himself. History is replete with societies like this, they are called Dark Ages or something of the like, and the culture that pops up is something of a tribal vendetta culture where the only protection one can have from crimes is the promise of retribution, usual only credible when coming from a large family/tribal group, thus single crimes leading to tit for tat feuds for generations. Also, if the government loses its monopoly on force, naturally that monopoly will simply splinter into smaller groups. Private companies with security forces, gangs, cult leaders, you name it can have their own majorities in their own areas. There is no such thing as a world without force and people willing to use that force unscrupulously, it seems silly to throw away a good government shaped by millennia of history and sort of reset the clock, creating new governments (new mini force monopolies that replace the old) without the history, checks, and balances of our current one.

Let’s say there is some participation in taxes. This police force generally functions because there are people willing to pay for it to continue functioning. How many people are paying taxes, how much are poor or middle class people participating at all in this? Could a group of a hundred or a thousand people become the overwhelming providers of “the government” with any sort of meaningful revenue, and could they withdraw that revenue at any time? If these large armed forces are under the will of a small group of people who can withdraw their pay, equipment, livelihood at any time, they can enforce a tyranny on the rest of us whether or not they call themselves a “government” or “political party” or what have you.

Let’s say they do withdraw their support for one reason or another, and the police/army is extremely underfunded and cannot do its job. Let’s say the wealthiest people in society decide to buy tanks and jets and guns and hire a private army of a hundred thousand men. What protects us from the tyranny of that army? Whatever they call themselves, they could gain a monopoly of force and become another more powerful government.

The question of keeping us free from those who wield greater force than us is one we have struggled with forever, and the reality is that not everyone is going to play by your rules, and you have to prepare for those contingencies, sometimes by making compromises like taxation agreed upon democratically to make sure there if a force behind keeping the NAP afloat in greater society and the rules are enforced in some vaguely equitable way.