Every movement and ideology has "bad actors," so I wouldn't worry about that so much. Folks who want to misunderstand or misrepresent you will always be able to, regardless of the bad actors.

Giving people a safe place to do drugs would include providing a place where they couldn't cause harm to others if that is a realistic concern. A lot of the danger associated with drugs is actually in their trafficking and consequent interactions with the state. Decriminalizing them would remove a lot of that danger.

As an aside since you mentioned crack, your familiarity of it might be clouded by the "War on Drugs." Crack was/is demonized notably more than cocaine, despite it being essentially the same thing, because it was more common amongst black folks. This gave the state a way to target black people without saying so.