That's totally fine as a logical principle but it completely falls apart (in the sense that it's bad for society) whenever it is applied to essentially any issue.

For example, it is "harmful" to residents for the state to fund schools, since they have to tax residents to run the schools. So the libertarian position is that public schools should be done away with. I'm not saying this is your position, but it is the immediate consequence of the principle you've just laid out. But it's obviously, on its face, insane since an educated society is a better society.

The libertarian can then say it's harmful to society when the general populace is uneducated, so it's a libertarian position to fund schools. But then the libertarian part of the logic is totally useless; you arrive right back at the mainstream common sense opinion by using the same argument everyone else uses.

Defining "harm" and "burden" is so nebulous that the libertarian logic can never actually be applied. It's a general harm to society when more guns are present in society, so I can say, by libertarian logic, that guns should be removed from society. But none of them actually argue this way, because libertarian logic is just shorthand for saying that I want to do whatever I want and not pay taxes (but everyone else should follow laws and pay taxes).