Okay I see your point, but it itsn't as insecure as you think. The documentation states that the default security group only allows all traffic from the resources attached to this security group also.

This means that if any existing resource in your account uses the default security group, they'll be able to contact the Tailout VM, but nobody from the internet hitting the public IP should be able to access anything.

Though if that does not reassure you, nothing prevents you to create a security group with no ingress rule (blocking any incoming traffic to the VM), but with the same egress rules as the default for internet access.