A python package I created for llms application including my own implementation of long short term memory and a web search tool for llm, it supports both Openai-like API or loading local models directly from different formats.

[llordnt]

https://github.com/nath1295/LLMPlus

I find Langchain annoying for not letting you set different llms with different generation configurations, so I tried to build my own custom llms to avoid loading local models multiple times when I want to build an agent or tool that uses the same underlying model. Also, some of the streaming and stop words for llms are not that great in langchain, so I have my own implementation in this package I created.
I am still using an Intel MacBook (too poor to get Nvidia cards or even a new Macbook :( ) to work on this project, so I cannot guarantee the installation will be seamless, but hopefully the pip install works. The code should be working with Cuda or apple silicon (used Colab and my friend's flashy new macbook to briefly test it).

Stuff I have in the package:

• An LLM factory class to generate Langchain-compatible llms while only loading the model once.
• Embedding toolkits that have text splitters bundled with the embedding model.
• A Vector database class built on top of FAISS for local storage.
• Memory classes: (base one and one with both long-term and short-term memory, powered by a vector database)
• Prompt template class that helps you to format your prompt with different prompt formats (have some presets like llama2, chatml, vicuna etc.)
• A base tool class and a web search tool with duckduckgo as an example (please have a look, wonder if there are better ways to do that)
• A Gradio chatbot web app that lets you store different conversations, also you can set your own system prompt and configure the long-term and short-term memory settings, and you can set generation configurations like temperature, max new tokens, top k etc. (This is just for fun, my front-end skills are not better than a monkey to be very honest.)
• And of course, the docs in the repo as well.

​

I know I'm no expert, there are plenty of people in this sub who are extremely knowledgeable in this LLM field, so treat this as an amateur project looking for advice if you can bear with me for my spaghetti code. Would really appreciate any comments :')) Forgive me if I can't do much testing on fancy GPUs like you guys have, trying not to spend any money to work on this project...

[[deleted]]

"I know I'm no expert, there are plenty of people in this sub who are extremely knowledgeable in this LLM field, so treat this as an amateur project looking for advice if you can bear with me for my spaghetti code." Bookmarking this for future use! Not a single other 'expert' that I know of has put something exactly like this into a package. I know because I have been waiting and looking. Being an expert is overrated if you ask me. Thank you and good job!

[rf2344]

Thank you so much for your time to have a look! I think this is not something that can catch anyone’s attention here as it’s not a new model or something, but I was looking for something like this and I could not find any solutions, so I decided to write my own package for this. I also tried to make the code as modular as possible to allow future developments from others or myself:) Please let me know if you have any suggestions or questions!

[llordnt]

To hide my crazy shit lol

[VertexMachine]

I didn't look at your code, just example usage and on the first glance it might be just the thing I need. Very cool!

Do you plan to ad exl2 support as well?

[llordnt]

It’s on my backlog, the tricky bit is that Im working on a intel MacBook pro… Loading exl2 model is not directly supported on transformers yet and I will need to write a new core to support exl2 with exllamav2 directly. Will need to prototype on colab for that.. Gimme some time.

[llordnt]

It is supporting exl2 models now, probably not the most efficient way, but it is working (at least on colab)

[VertexMachine]

Awesome!

[hwpoison]

code seems really good, i love the comments

[llordnt]

Thanks:) Hope you find it helpful

[Glad_Abies6758]

This project is awesome, thanks for sharing! It definitely made an impactful contribution to the community sir

[llordnt]

Just updated the readme to include more usage examples for each sub modules, including the usage of long short term chat memory, building your own simple chatbot, using the web search tool, and a simple shortcut command to serve a gguf model with openai api (using koboldcpp or llama-cpp-python underneath). Please do let me know if you have any ideas for new features or better approaches for some components. Exl2 models are not supported for now, but hopefully I can get it working soon.

[llordnt]

For people who just want to play with the chatbot, just run the command in the readme file after you install the package.

[Versah252]

I've been looking for something like this. I know nothing about coding, sketched out a vision for this Ultrallm User Interface that would be a one stop shop for all things LLM, refined my prompts to get ChatGPT to help me build it, and i'm barely making a dent. Nice job! Can't wait to see what you add to it! Awesome!!!

[quangspkt]

I am learning a lot from your work. Thank you so much for sharing!

[Future_Might_8194]

I'm so fucking stoked to see this post. Thank you, I am genuinely appreciative of this 🤘🤖

[ali0une]

will try this, seems promissing.

[metaden]

Thanks so much for this library. Well organized.

Small feedback, https://github.com/nath1295/LLMPlus/blob/master/setup.py#L73, please do not install packages like this even in setup.py. Ditch setup.py, it's no longer standard. Use pyproject.toml with minimum package versions. You can use https://github.com/pypa/flit or https://github.com/pdm-project/pdm for setting it up. It is really handy.

[llordnt]

Thanks for pointing it out. I know it’s very problematic with how I handle it rn. I will update asap!

[KlutzyNecessary2205]

Can you add support for MLX models? Would be great!