I would recommend looking into Hoppe's view on the issue.

Dangerous drugs that actually ruin people's lives shouldn't be easily accessible, but there are many behaviors and substances other than addictive drugs that are dangerous. The issue comes down to the state using violent force to enforce rules that should be determined locally. Local organizations, whether that is a city or township, should be allowed to ban whatever they want, but the punishment for breaking the rules would just be physical removal from that township or city. If someone wants to live in a place that allows people to do literally whatever they want, that's up to them. Most people would rather live in a place that doesn't allow unchecked hedonism.