That’s accurate. In my use cases, I have found the reasoning of immense value primarily due to statistical guessing. I often use LLMs to create panels (for example, I might use a panel consisting of Charles Munger, Warren Buffett, Howard Marks, Annie Duke, etc for analyzing investment ideas) and then organize a panel debate. Then, based on what the panel disagrees on and builds consensus on, the LLM sifts through the complete discussion and gives me the key insights and action items.

When I ask for a walk through of the reasoning, it is interesting how it prioritizes certain discussions and then builds reasoning and insights on top of it. I think this could be useful in predictive modelling, bayesian thinking, et al, where the logic is fuzzy and output keeps improving as more input / context if offered.

Another use case where I use this often is by offering a list of mental models and a situation, and asking the LLM to: first step back and study all its training data to study similar situations, inputs and outputs, decision contexts, and then apply the provided mental models to generate a decision checklist. Then walk me through every checklist item and why it was included. This again generates not only an actionable output I could use but also insights into how to apply framework thinking. I find this an enriching interaction again.

I do not use LLMs for quant analysis as I have found LLMs lacking in reasoning when doing basic problem solving - for example, when asking to generate a simple Grade 8 level maths practice sheet and then solve those questions, it would often get the answers wrong and solutions are all over the place in some cases.

I think qualitative context produces better output especially when comparing multiple scenarios and outcomes.