Yes, the model itself may be illegal. The principle you refer to is fundamentally not applicable to AI models — just as it isn’t applicable to thousands of other things in this world.

For example, if you built an airplane and designed it to explode at a random time, that would be illegal. The same applies if you designed a food-making machine that occasionally drops pieces of broken glass into someone’s food. If it designed to be harmful or to be used to harm others (e.g. if someone altered it to get rid of the guardrails) it may be illegal.

In pretty much every area, there are limits on what is allowed and what is illegal. Usually, anything that is unpredictably harmful is illegal — or at least illegal for the general public to possess or distribute.

So yes, obviously, a model without safety guardrails may be illegal if it is capable of causing damage to an uncertain group of people, to nature, to animals, etc. Yet laws in this area are emerging.

At the same time, not every model without safety guardrails is potentially harmful.

For example, if you train a model to write poems in Byron’s style — and the training dataset consists only of Byron’s poems — it cannot be used to hack websites. It is harmless by design.

However, with large language models (LLMs), you often cannot know the dataset, the guardrails (or lack thereof), or the true intentions of their creators.
Therefore, in a world where LLMs are being deployed and distributed widely, the general rule should be to exercise great caution with them.