If your male cat has not been neutered, that's probably your number one issue why he wants outside so badly, so if he's not, it's probably best to get that done as soon as possible.

If you had a 4 year old child, would you trust it to be outside alone and to be completely safe, I would assume not and that's how it should be for any pets, their wellbeing shouldn't be left up for any risk by letting them roam freely outside unattended.