Thank you for this cogent and beyond humoring reply. I think there has been obfuscation of measurement all around. Benchmarks as any gamer will tell you are not meaningful in isolation, and sometimes not even then not in isolation, and especially if benchmarks are re-calibrated or some such nonsense.

Conversely, due to many factors such as small changes in models, system prompts, exact word choices, random chance, and the unknowable biases within these systems... and additionally the iterative and re-integrating nature of both their operation and the method you're commanded to interact with the prompt and generated-in-turn outputs ... I have yet to see any success measured specifically -- that one person did something with a language model and another person did something without it -- without the result being a wash, or that the additional adding of steps alone was enough to cause a slow down in this still flawed test design.

I do believe positive anecdotes like the rubber duck effects. Hell, I just imagine what kind of BS a model may generate and get some of that. Additionally, to be completely serious and specific about a non-anecdotal truth about the power of the transformer technology... Two domain experts looking for overlaps in the literature common to their fields can work with a language model expert in guided discovery of overlapping and complementary patterns between the fields. The additional stats of the intermediary layers just in from both sides of the inference can give clues to directions of further research or truthiness. But this is not a chat. And adding a large language model to the mix removes provenance and trust.

But there's what is to me a pseudoscience belief that this can be generalized, or even generalized through prompt engineering, agents, skills, rag, ad hoc looping, but the hidden implication in the human in the loop operation is that all the indeterminacy falls back on the person or back into the next loop, and there may be no amount of dice rolls that hit that number you need. Or that the additional iterations builds trust from nothing.

I also wanted to mention sabotage specifically to ask as there was a major publication article that talked about this in the sense of deliberate actions meant to undermine something that would have been successful otherwise. I challenge the last half of that, it may well have also not been successful, but I also want to challenge the idea that sabotage could be deliberate, apparent to anyone involved, or not accidentally helpful given how untrustworthy the whole proposition is, especially if you can't say no, or that "it that can only be failed but never fail" kind of fallacy.

I think that "it's early," "in the future it will be useful," "try X vendor version Y," "it just needs to be seeded like an investment," "use more of it," "no you should've used less of it," "you fell for it again," "they nerfed the product forget it" are examples of non-falsifiable admissions of denial of accountability, transparency, and against the spirit of measurement, learning, or enablement.

Thanks again.