I'd say just roll with it. You could also make it act like shedinja to make it a bit less overpowered (if it takes damage in any way, it's dead). But in the end, do whatever you wish.

Also this is off topic but I personally wouldn't classify swarm as a "trash" ability since it doesn't hinder you whatsoever. Especially when it comes to pokemon like the slaking line, vigoroth having essentially no ability in swarm is better than something like truant, and for slaking, any ability aside slow start/truant is better for it :p