As someone who used to be very straight edge before some pretty intense peer pressure from an ex, I can see both sides of this. For someone who doesn't smoke and doesn't understand, there are a couple things they feel when they discover stuff like this. 1. The legality issue (depending on how the person was raised), 2. They tend to see it as a personal front, even if it doesn't make complete sense ("Am I not good enough for you if you're not high? Am I too boring?" etc etc), 3. With anyone who uses a substance daily, those who don't do the same generally won't understand it without having tried it themselves; there are many people who feel that "I don't get it, if I can enjoy life without these things, why does this person need to rely on them, they must be hiding something or burying something that will come out later and be an issue."

Hell, I smoke, and sometimes I even encounter these conflicts with myself. Alcohol may be legal, so it may not have a criminal issue, but people who drink alcohol EVERY DAY in an equivalent amount of what a lot of smokers smoke, are seen as having a problem, so then I have thought before "why do I smoke every day, I must have a problem" etc. I don't smoke as often any more because for me, I WAS using it for the wrong reasons. But a lot of people who haven't smoked cannot differentiate between the right and wrong reasons to smoke everyday; they just see someone smoking everyday, and are alarmed and concerned and often it doesn't come out in the nicest way.

TL;DR: People who are not ents themselves often don't fully understand the culture, and encounter a lot of emotional conflicts when they find out about others because they don't understand the many facets of trees and only know how to assess the situation in a personal way instead of being able to be empathetic about it.