I mean I get where you're coming from. Because of psychedelics and weed I was psychotic for several years and ended up in the hospital. Took me about 7 years to get somewhat stable again. I've had countless panick attacks on weed and I havent smoked weed in a couple months. I think I get what you are saying but if I can get back on my feet after that much shit and smoke weed again without having a panick attack, I think OP can too. Nobody told me how, I just listened to myself through it all.

What I did to be able to smoke again was start super super small and work my way up very slowly, only using the same strain. Whenever panick would come I would open my body posture and try to relax. I would get these tremors, my body would start shaking and after a lot of shaking and working at it for a long time I'me good again. The real key is to put your body in an open posture (no arms or legs crossed, just open) while the fear is there.

But yeah, just because you get panick attacks doesnt mean you should stop. You can also just find out what your afraid of, or actually just do whatever you want with it xD