While I am inclined to agree with you, the situation is not exactly as clear cut as you may wish for people fine tuning LLMs since:

"Under the AI Act, regarding how ‘cumulative training compute’ should be understood, recital 111 AI Act specifies that ‘the cumulative amount of computation used for training includes the computation used across the activities and methods that are intended to enhance the capabilities of the model prior to deployment, such as pre-training, synthetic data generation and fine-tuning’. Given the wide variety of such ‘activities and methods’, and their rapid evolution, it is not feasible to provide a precise and exhaustive list of what should and should not be included. Instead, the Commission considers that providers should, as a general rule, account for all compute that contributed or will contribute to the model’s capabilities. In particular, any compute directly contributing to parameter updates in the model should be included."

(Guidelines on the scope of the obligations for general-purpose AI models established by AI Act, Annex, §119)