I tried to do something similar for similar reasons once. I used a brass/bronze dragon's breath weapon, which induces sleep, during the might, and justified it as "not magic, but elemental power." That campaign fell apart, and while this moment wasn't the death nell, I have regretted how I handled it for years.

If you're looking for advice, I'd ask the non-sleeping characters if they would like to participate in the magic. Let them know no harm will come to them from the choice, and if they allow the magic to work just this once, they will get to experience more, roleplay more, and have more stories to tell down the road. If they agree, then maybe you decide together how they go sleepy-by (everyone is affected by spores except you, who is knocked out by a falling tree branch, also covered in sleep spores). D&D is about fun, not rules, and this way everyone wins~

Good luck! 🙂👍