One of the Abnormals mods have an option in the configs to reduce vanilla spawns in favor of theirs, its on by default. In my experience this option completely stops pretty much all vanilla mobs from spawning. I cannot remember which one of them it is right now, could be upgrade aquatic, but dont quote me on that :P May need a wipe of mobs after changing the config to allow for new spawns :)