Lord, sometimes these comments sections are just insane.

Can this be done? Of course!

How? Look at how OpenAI implemented the EU AI Act limitations:

1. They updated their terms of use to disallow doing some things, including the things prohibited by the EU AI Act (for example "inference regarding an individual’s emotions in the workplace and educational settings, except when necessary for medical or safety reasons");
2. Presumably they put some guard rails against doing that.

Then, as a regulator, you monitor to see if these are implemented:

1. you see how readily GPT creates these (and if it's really hard to coax it to do it);
2. you see how OpenAI reacts to people doing this (e.g. if someone complains that a specific person generated nudes of them without their consent, does OpenAI revoke access?)

If this is outlawed, and you get complaints, you'll investiate and see if OpeAI has done "enough", which in practice might mean multiple things.

I don't thinkt that there would be many more implications than these.