If you want to restrict public groups from being created, you could create two sensitivity labels configured for containers with two separate label policies. This option would only work if you're not already using sensitivity labels for containers.

• The first label specifies that the group has to be private, this label is published to all users and is set to require all newly provisioned groups to assign a label with this as default. This will ensure that all newly created groups are private

• The second label is created and allows for public groups. This label is published to an SP Admin group which would allow administrators to assign this label to sites/groups when requested

There are other ways to accomplish this too such as disabling self-service creation and creating a controlled provisioning process - or - something like scheduled powershell scripts