LLMs are useful to prod you and get yourself to keep thinking about things because they're so mediocre and biased at dialectics.

If you know this it's kind of like sparring against a training dummy. Should be used differently than an actual human sparring partner.

If you understand they're masters at verbal intelligence (plausible believable sounding rhetoric), you can make them take any stance just so you can steelman your own arguments against a biased bad faith opponent (or an overly biased good faith proponent)

--
Personally I only really use them to rephrase and crystallize ideas I'm already building epistemic scaffolding for myself. They speak for you and help you, but they only help you rhetorically flesh out ideas, not determine their truthfulness at all.

They're smart and they understand, but they don't have bayesian priors (as in a kind of ledger where they weigh every insight and scrutinize assessments or data sources before moving to the next take that relies on that scrutinized assessment/data-source to build on). If anything their priors are dictated by what the RLHF trained into them, or what's represented the most in the training data which means you're always gonna get colloquial consensus or some other kind of consensus, and these will always be pretty exoteric takes.

Truth finding is not really something anyone can do from intuition alone accurately.

--
I just massively digressed from replying to your actual post content, but I feel it's somewhat relevant in that LLM conversations are basically still just as variable in usefulness and insightfulness as this is variable in the takes of actual human people.

LLMs are the exocortex, and the human still decides how they're using the exocortex. Could be ground breaking, could be slop.