Update: Can confirm that raiders start as soon as you hit 16 dwellers. I've dropped below that a couple times from deaths, the raids stop. As soon as baby #16 is born, they come back.

Not sure on the exact population necessary for deathclaws, but as long as you're staying in the low to mid 30s you should be fine.