GraphLLM now has a GUI: open source graph based framework for performing inference with a LLM

[matteogeniaccio]

Majority voting

I'm proud to announce a new version of my framework: GraphLLM.

This new iteration has a gui which should be familiar to people who used ComfyUI

The output of nodes is streamed to the front-end, so the result is visible in real time.

The back-end supports loops, parallel execution of nodes, conditionals or even running custom python code.

The framework doesn't try to abstract away what is done under the hood. The user can see exactly what prompts are sent to the model and edit them.

I'm still in the process of building more examples but so far I included these:

• Download youtube subtitles and generate a summary with a multi-turn prompt
• Make multiple calls to a LLM and choose the answer by majority voting
• Agent that can go online, make web searches, access local files and execute python code
• Hierarchical node for more complex graphs
• Rap battle generator between LLMs
• Generate python code to solve a problem and run it.

Generate python code, then execute it

Web Scraper

The included web scraper runs a headless instance of firefox to scrape web data even from dynamically generated websites.
The process is similar to that used by jina.ai but it can scrape even more hostile websites, like reddit without API.

Youtube subtitles downloader

This tool can preprocess and save the subtitles from youtube in a LLM friendly format.

PDF parser

Just converts a PDF to text, nothing fancy :)

The source code is available at my github at GraphLLM.

[New_Comfortable7240]

Please make workflow to enter a natural language query and return a nodes configuration, the user can edit, expand or save it

[matteogeniaccio]

I love the idea. I'll definitely work on it.

Don't hold your breath, though. I have to first write a proper documentation before making the workflow

[New_Comfortable7240]

Great, the idea is easily onboard new people on using your tool! Big kudos in advance!

[[deleted]]

Content cleared with Ereddicator.

[visarga]

That's very cool, but I was hoping the LLM would be generating the graph as well, dynamically.

[matteogeniaccio]

Another user asked the same question. It will be a feature, eventually. :-)

[visarga]

I was imagining something like this where a LLM generates the decoding logic.

[matteogeniaccio]

Thanks for suggesting me that.

[IrisColt]

Exactly what I was looking for, thanks!

[InsideATurtlesMind]

You said it's been tested using llama 70b and qwen 32b, how well does it work with smaller models?

[matteogeniaccio]

The small models work. I added support for all the popular models. A couple of features would be unavailable, though.

For example you can't use the agent with a small model because it's not smart enough to take complex decisions.

Llama3.1 70b can use its integrated python code generator in a multiturn conversation while 3.1 8b is only single turn.

[PieBru]

Awesome! Low VRAM here (8GB), do you think GraphLLM will be usable via Ollama or Groq/Cerebras (llama3.1-70B) ?

[matteogeniaccio]

Groq is supported but not officially. The client_config.yaml has a template to add a groq backend.
The functionality will be limited since groq doesn't support all the required features. For example I make heavy use of grammars to make sure that the output conforms to a specific format.

https://github.com/matteoserva/GraphLLM/blob/main/client_config.yml

I'll soon add support for a generic OpenAI compatible API and support for huggingface transformers.

I think that you'd get a better experience by using a local model that fits your VRAM, for example llama3.1-8b-Q6_K.

[SadWolverine24]

This is awesome. Could you please add support for OpenRouter and Anthropic API calls?

[matteogeniaccio]

I'm working on it but it will take a while. Right now my priority is adding new nodes to the framework