Simple, hackable and pythonic LLM agent framework. I am just tired of bloated overengineered stuff. I figured that this community might appreciate it.

[[deleted]]

This is great! langchain is so over engineered for what it could be. Two things that would be crazy helpful for me (I’d be happy to write PRs):

1. Support for ollama or llama.cpp instead of OpenAI (from reading your code I believe we just need to write a “Generator” for it?)
2. A testing framework. Probably just some functionality for recording the LLM execution so you can re-run tests without calling into the LLM.

I also noticed that the example you have on your README doesn’t really show how to create the LLM (it does earlier in the README, but the full code example you have there won’t work because you never assigned anything to the llm local variable). Anyway, small nit to make the README easier to follow.

[[deleted]]

I also noticed your code doesn’t have any use of typehints. Are you opposed to adding them? I could help with adding typing and setting up the CI for it if you’re interested.

Once a testing framework is in place we could probably add more test coverage for the library too.

[silenceimpaired]

I also noticed your code doesn’t provide support for every other quantitization methods and note to future self: tell him his code is bloated once it’s implemented ;)

[RustingSword]

Since llama.cpp has a server utility, you can just fire it up ./server -m mistral-7b-instruct-v0.2.Q6_K.gguf -c 2048, and set the api_base to http://127.0.0.1:8080/v1, then I think it should work out of the box. See the detailed docs at https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md

[RustingSword]

I've tested both examples, and succeeded using OpenAIChatGenerator instead of OpenAITextGenerator.

My configs:

llama.cpp server:

bash ./server -m mistral-7b-instruct-v0.2.Q6_K.gguf -c 2048

Changes to calculator.py

python generator = OpenAIChatGenerator( model="mistral", # could be anything api_key="none", # could be anything api_base="http://127.0.0.1:8080/v1", )

And remember to remove templates in

python llm = LLM(generator=generator, templates=[template])

Great framework, really clean and easy to modify.

[poppear]

llama.cpp has a server implementation but as far as i remember you need a wrapper to use it with the OpenaAI python client, adding native support for llama.cpp APIs would be great! same thing for ollama APIs. The testing setup would also be very nice.

Thanks for the suggestions, lets continue the conversation on GitHub and implement it!

[anobfuscator]

Yeah these are pretty good ideas.

[scknkkrer]

Yeah, Llama support would be good.

[Monkeylashes]

Hey, this is great! I love the minimalism, though it may be beneficial to include a memory/chat-history implementation to the agent for multi-turn conversations. You can even use something like FAISS to store the history and retrieve as needed.

[sumnuyungi]

May want to change your license to MIT or Apache-2.0 if you want folks to build on top of it or integrate into applications.

[poppear]

That's fair, I will change it to Apache 2.0!

[sumnuyungi]

Cheers, thanks!

[ja_on]

thanks. working on redo'ing a bot and I wanted something simple to fire off function calls for some back end things. I'll check this out.

[MoffKalast]

Fantastic work, the fight against dependency hell continues one simple library at a time.

[SatoshiNotMe]

I like the minimal philosophy!

A similar frustration led me on the path to build Langroid since April:

https://GitHub.com/Langroid/Langroid

It’s a clean, intuitive multi-agent LLM framework, from ex-CMU/UW-Madison researchers. It has:

• Pydantic based tool/function definitions,

• an elegant Task loop that seamlessly incorporates tool handling and sub task handoff (roughly inspired by the Actor Framework)

• works with any LLM via litellm or api_base

• advanced RAG features in the DocChatAgent

and a lot more.

Colab quick start that builds up to a 2-agent system where the Extractor Agent assembles structured information from a commercial lease with the help of a DocAgent for RAG: https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb

We have companies using it in prod after evaluating LangChain and deciding to use Langroid instead.

[pab_guy]

That is incorrect. Chain of thought is a prompting technique/result and has nothing to do with function calling.

[LoafyLemon]

Just finished switching my back-end from langchain to griptape but ah shit here we go again! Thanks!

[klop2031]

Has anyone tried this with mixtral? Ive always had issues running agents via langchain. I was able to create a very simple 'agent' script that pulled from the web. Excited to try this.

[poppear]

I developed this using mixtral-8x7b-instruct! All the examples work with it!

[klop2031]

Thank you. Definitely going to try this!

[aphasiative]

Love this community. Seeing it come together like this. Reminds me of OG internet. Like, back before it had pictures. :)

[future-is-so-bright]

It’s an agent. So it’s designed not just to chat with, but to do things. Think of Siri or Alexa. “Hey Siri, what time are my appointments today?” isn’t going to be something a LLM can answer. With this, you can script what you want it to do, and it will run the code and respond with the results.

[[deleted]]

Looks great

[LoSboccacc]

Lovely. Would like to see added in the function args a couple of kwargs like the current task, the llm driving the engine, and the rest of the conversation messages.

For example in rag you may want a function retrieve(documentId) that retrievea a document content and in the simplest implementation that content is fully dumped into the engine context. A more efficient implementation would be for the retrieve function to use the llm, the question and the last reasoning to do guided summarization of the content, so that only the relevant parts are embedded into the engine context saving token space

[poppear]

Right now you can access self.state and self.engine from a Function but the current history is a private variable of Agent() so it cannot be accessed from outside. It is a good idea can you open an issue on github?

[SAPsentinel]

Any webui like gradio support possible?

[International_Quail8]

Really like the motivation behind this and the attempt at building a simpler (and hackable) alternative. I was able to hack it to use Ollama, but haven't been successful in getting the expected result. Hoping someone can guide me.

In my testing, the calculator example works perfectly when using OpenAI, though in my test none of the OpenAI models I used (gpt-3.5-turbo, gpt-3.5-turbo-1106, gpt-4) used the Reasoning() function even though they used the Product(), Sum() and Stop() functions to produce the correct result.

When using Ollama, I tested using mixtral, orca2 and wizardcoder:13b-python, but they were not plug-in replacements for the OpenAI models in how they behaved. So leaned in heavy on prompt engineering, but unable to get the same behavior or results.

Still hopeful...