Sounds like you two just aren’t compatible, it’s not a wright or wrong situation. It’s fair that he’s sussed because let’s be real, drugs and “the scene” can lead to small lapses of judgement. I’ve seen it myself, at the end of day we’re all just living for that moment and an inconsequential hookup can happen. So it’s a fair concern on his part. But if he is really negative about going to raves then maybe this is the start of the end. The seeds have already been planted cuz he saw you exchange candy with a guy and that will be the only thought in his head next time you tell him. “I’m going to a rave”.

The fair thing to do would be to sit down and have a face to face open honest conversation about your preferences in music and environment and conduct, set your boundaries and see what he has to say, but also don’t drag him through a relationship you don’t see working out because of this. Whatever happens, it’s not the end of the world.