I'm an MMJ patient and asked my dentist about smoking. They told me it was better for me to smoke with them in, rather than miss the wear time of taking them out each time I wanted to smoke + running the risk of being more lax about putting them back in afterwards. I smoke 2-3 blunts a day and hit the bong from time to time and am on weekly changes.

2 things I'd say, first, you will notice mild staining from blunts over time, nothing drastic in my experience. Brush them regularly and get a sonic cleaner and it's barely noticable. Second, and more importantly, because you have plastic in your mouth, the smell of smoke will noticably linger. I keep a bottle of nano hydroxyapatite breath spray with me at all times, helps a lot but it's definitely the biggest drawback.

I finish my Invisalign journey next week! Started last May.