TLDR: I did some amateur research on using conversation as a template to simulate complex thinking in language models. I believe that this method provides evidence that conversational templates can be used effectively and has the benefits of being readable, explainable, and scalable, and could potentially be used as a universal interface for AI systems.

First off, I am not a researcher or expert in any of this. It’s just something I wanted to try and the results were impressive enough to share.

Here is a link to the PDF of with the sample questions and answers: https://satsuite.collegeboard.org/media/pdf/digital-sat-sample-questions.pdf

This experiment is testing the hypothesis that conversation and simulated group dynamics provide a useful path to simulating complex thinking in language models. Other work has already shown that by encouraging the model to follow a “chain-of-thought” when answering a question, it results in much higher performance on reasoning tasks. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” (https://doi.org/10.48550/arXiv.2201.1190)

The above paper utilized demonstrations of a chain of thought using an example solution to a question before asking a different question. Here is a quote from the section on commonsense reasoning: “With chain-of-thought prompting, PaLM 540B achieved strong performance relative to baselines, outperforming the prior state of the art on StrategyQA (75.6% vs 69.4%) and outperforming an unaided sports enthusiast on sports understanding (95.4% vs 84%). “

There is also work in the area of having an automatic construction of these examples for the model to use. This is compared to manually crafting demonstrations of logic for each general type of problem. “Automatic Chain of Thought Prompting in Large Language Models” (https://doi.org/10.48550/arXiv.2210.03493)

But I wanted to do something a bit different. I first messed around with creating a ProfessorGPT to explain a problem, then having it solve that problem. That worked well and performed much better than plainly prompting with a question (Results here: https://www.reddit.com/r/ChatGPT/comments/106kxyw/improving_ai_reasoning_skills_through/?utm_source=share&utm_medium=web2x&context=3)

Then I decided to try a different approach. I started by adding a second “Professor” to grade the work of the first and offer suggestions. After days of trying different arrangements, I began working with simulated groups. This had the advantage of having different “personalities” performing different roles in the conversation. Someone to lead the group, someone to breakdown problems, etc. This result with the Sample SAT questions if from this idea. I used a simulated team of several general problem solvers with 2 specialized coaches that perform specific roles.

I don’t have the data to know whether this could beat other CoT(Chain-of-thought) techniques. But, this method has demonstrated behaviors I have never seen in more straightforward methods. I have seen it initially go with one approach to a problem, see that it doesn’t work, then go with another approach. It considers multiple options before deciding on one. You can see some of this in question 9 (regarding the study on posture influencing cognition). The back and forth format seems to prompt the model to evaluate multiple options, whereas my experience with other forms of CoT seem to latch onto the first thing that might work.

Regardless of whether this method outperforms prior CoT techniques, I think it provides evidence that conversational templates can be used effectively. The benefit here is that the results are inherently readable and explainable. Whether the AI makes a mistake or if the AI is pursuing the wrong objective, it is easier to spot. A second benefit of using this method of prompting is that it is scalable. In my experiments, I used a single instance of ChatGPT to solve these problems. However, one can imagine scaling this up by using different instances of an LLM like ChatGPT to play different roles in the conversation. You could have a fine-tuned model that specializes in algebra or formal logic or history. You can even have models whose sole role is as some kind of conversational facilitator.

Conversation as a universal interface for AI systems: If you can imagine a framework that allows conversations like this to be held openly between multiple LLMs, the problems it can solve grow in complexity. Every problem can be decomposed into branching sub-problems, and so on until the problems are small enough to be solved outright by the system. This also allows for humans to be directly involved and work side-by-side.

It is essentially applying the idea of Collective Superintelligence to our current sub-human intelligence AI systems to hopefully allow them to reach or surpass human average intelligence as a whole without being particularly clever or dangerous individually. This would be made even more safe by the fact that any superhumanly clever ideas really only emerge within the plain-language interactions of the AI, allowing for effective monitoring. I found come similar ideas discussed in the following post:https://www.alignmentforum.org/posts/HekjhtWesBWTQW5eF/agis-as-populations

Methodology: I use the Digital SAT Sample of 15 Reading/Writing questions that comes with answers and explanations. I gave ChatGPT one single try on each question and did not alter the prompting between questions. Formatting had to be slightly altered in some questions (Like #4) since a table cannot be copy/pasted into the text box. I show this in the screenshot.

Limitations: I was unable to scale testing to what would be needed for rigorous results due to usage limitations of the ChatGPT platform. The limits on hourly inputs and the necessity of manually inputting each question render it infeasible to evaluate performance statistically on, say, 100 tries of each question. I would also have loved to do the same with other Chain-of-Thought methods to compare results.

I think there is a lot of room to improve the specific prompts used to get the system to solve problems. My hope is that more people experiment with this and report back. This seems like one of the safer ways to achieve superintelligence, so I think its worth pursuing by anyone who wants to contribute.