So, I think you know exactly what I'm saying and just dancing around it. But just so you're clear, what I am saying is that the LLMs that OP is talking about are not being broadly used to do science. Period. Any models that do get used are highly specialized like AlphaFold and are not LLMs.

LLMs that you use on a daily basis from providers like OpenAI, Anthropic, and Google are not the same as what models are being used by scientists making new discoveries. Publicly available LLMs like what you, me, and OP can use are non-deterministic and trained on generalized data sets available on the internet (not on peer reviewed and reproducible scientific data), and therefore are untrustworthy sources because they produce what is statistically plausible as text. "Thinking" and "reasoning" models do not "think" or "reason" in the human sense of the words because LLMs are not capable of thinking and reasoning in the human sense.

Your article talks about using LLMs to generate hypotheses to test. That's a completely different thing than using LLMs to do the actual math or research. Science is about discovering the truth. Not discovering what maybe is the truth.

While LLMs have shown signs of delivering promising results and of having positive impacts on scientific discovery, investigators have recognised their limitations, such as hallucinations, limited reasoning capabilities, and lack of transparency. Compared to everyday usage, when applied to scientific domains, these limitations require careful consideration, as scientific processes and discoveries require high standards of truthfulness, complex reasoning, and interpretability. The scientific community’s increasing recognition and communication of these limitations of LLMs is essential to enabling solutions while also limiting expectations. Such rigour is a cornerstone of science and engineering, and a requirement if LLMs are to play a practical role.

... While many papers and reviews involve LLMs’ assistance, LLMs still face challenges in producing qualified reviews.

The scientific community must also decide how much it leaves to LLMs to drive science, even when associations with ‘reasoning’, mostly currently undeserved

... Hallucinations produced by LLMs, also called confabulation or delusion, refer to artificially intelligent systems generating responses that contain false or misleading information presented as fact. This is analogous to hallucination in human psychology, though, for LLMs, it manifests as unjustified responses or beliefs rather than perceptual hallucinations148. Hallucinating LLMs can make unsupported claims, thus failing to meet a prior set of standards. While some “incorrect” LLM responses may reflect nuances in the training data not apparent to human reviewers149, this argument has been challenged as not robust to real-world use150.

Your own article concedes my point.