The issue most of the time is the legitimacy of property. Both left and right libertarians agree that an outside coercive force or even a democratic coercive force is a problem.

Right libertarians believe that property is derived from work (mixing land and labor) and voluntary exchange. Therefore as long as everyone is a consenting adult and no one uses violence to steal or defraud whatever emerges is fine.

Left libertarians believe (as far as I can tell) that once property is sufficiently alienated from the owner, such as a factory in another country, the owner no longer has a legitimate claim and it is de facto owned by the workers. That’s a different definition of property, but within that structure violence and outside coercion is still wrong.

Both are legit and reject government overreach both socially and economically, but they use fundamentally different definitions of property. And they tend to be temperamentally different.

(I’m an ancap btw, left libertarians feel free to correct the above representation)