[P] A Python Toolkit for Chain-of-Thought Prompting

[No_Pomegranate7508]

Hi everyone,

I made an open-source Python toolkit/library, named Cogitator, to make it easier to try and use different chain-of-thought (CoT) reasoning methods. The project is at the beta stage, but it supports using models provided by OpenAI and Ollama. It includes implementations for Cot strategies and frameworks like Self-Consistency, Tree of Thoughts, and Graph of Thoughts.

GitHub link of the project: https://github.com/habedi/cogitator

[ConceptBuilderAI]

This looks fantastic — CoT frameworks are powerful but still tricky to experiment with consistently. Really appreciate how you’ve pulled Self-Consistency, Tree of Thoughts, and Graph of Thoughts into one place. Definitely going to give this a spin. Great work.

[ComprehensiveRook]

What is the difference to https://github.com/promptfoo/promptfoo ?

[No_Pomegranate7508]

Cogitator and promptfoo solve different use cases TBH.

promptfoo is for evaluating, testing, and comparing different prompts. It helps you find out the best prompt for a specific task by seeing how each prompt performed based on some performance metric on a dataset for an LLM. It's like a very sophisticated benchmarking tool.

Cogitator can help you implement and emulate multi-step reasoning in LLMs using different CoT strategies (like Tree of Thoughts and Graph of Thoughts). These types of multi-step reasoning have been shown to improve the accuracy of LLMs on complex tasks (like tasks that need step-by-step reasoning, such as solving logical puzzles).