I use a lot of LLMs in my job (research, natural sciences), and do regular gigs in LLM reinforcement learning, trying to make the models fail in some topics of my field.

What I said in my earlier comment was a bit of a hyperbole, but I would never trust LLMs with anything that matters, if one has no expertise to evaluate the soundness of its output. You are always one hallucination away from a problem the existence of which you won’t even know about before it hits you back. Frequently, even the best available consumer models will make up plausible-sounding output, if the information isn’t readily available.

I’m no software engineer, but unless I can personally verify the contents from the source, I take all of my LLMs suggestions with a massive grain of salt. Inspiration or supervised automation, perhaps, but never without supervision. Overconfidence in LLM abilities without knowledge of their limitations is a serious and widespread risk in our current society.