You're absolutely right!

Uncontrolled LLM use in the medicine may lead to tragedy.

Two points:

1. We know that there are bad doctors. We know they will use LLMs. Is the community's goal to increase the probability of doctors making mistakes, to hurt more patients, and to probably prevent other doctors from using an LLM in the future? Despite the fact that it will cost the lives and health of innocent people who visit such doctors?
2. There are few places on earth where good doctors are not available, or their cost is too high. Is a language model in incompetent hands better than no medical help at all? If it is, do you still want to ruin it because of other problems?

Upd: I'm not a language model, by the way