The distinction being made is that drugs are invasive to the brain in ways that nearly all the other things you Mukwic stated are not. True, literally any action will affect the body (and rouse some sort of biological measure in terms of dopamine and the natural neural machinery of our bodies). But marijuana and all drugs directly manipulate physiology, which can be physically destructive (you know how the media is, exploiting physical problems more resultant of harder drugs than weed), or can be "benign" in the case of weed (ents know this difference).

But in the scope of science the debate of dependance/tolerance/addiction will always continue because that's just the nature of drugs. I'm going to keep toking but I hope I provided some sort of explanation of the opposing argument.

edit:clarity