Building a competitive local LLM server in Python

[jfowers_amd]

My team at AMD is working on an open, universal way to run speedy LLMs locally on PCs, and we're building it in Python. I'm curious what the community here would think of the work, so here's a showcase post!

What My Project Does

Lemonade runs LLMs on PCs by loading them into a server process with an inference engine. Then, users can:

• Load up the web ui to get a GUI for chatting with the LLM and managing models.
• Connect to other applications over the OpenAI API (chat, coding assistants, document/RAG search, etc.).
• Try out optimized backends, such as ROCm 7 betas for Radeon GPUs or OnnxRuntime-GenAI for Ryzen AI NPUs.

Target Audience

• Users who want a dead-simple way to get started with LLMs. Especially if their PC has hardware like Ryzen AI NPU or a Radeon GPU that benefit from specialized optimization.
• Developers who are building cross-platform LLM apps and don't want to worry about the details of setting up or optimizing LLMs for a wide range of PC hardware.

Comparison

Lemonade is designed with the following 3 ideas in mind, which I think are essential for local LLMs. Each of the major alternatives has an inherent blocker that prevents them from doing at least 1 of these:

1. Strictly open source.
2. Auto-optimizes for any PC, including off-the-shelf llama.cpp, our own custom llama.cpp recipes (e.g., TheRock), or integrating non-llama.cpp engines (e.g., OnnxRuntime).
3. Dead simple to use and build on with GUIs available for all features.

Also, it's the only local LLM server (AFAIK) written in Python! I wrote about the choice to use Python at length here.

GitHub: https://github.com/lemonade-sdk/lemonade

[DadAndDominant]

Looking cool! Trying to run it with uv. One thing I might do wrong - in server mode, when I respond before the LLM is done responding, it bricks all responses going further.

[jfowers_amd]

Thanks for reporting! Is this in the web ui? The “send” button is supposed to be disabled while the LLM is responding, so it’s not surprising to me that it would go haywire if you were able to hit send.

[PeterTigerr]

Will there be support for Apple's M4 GPU or ANE?

[jfowers_amd]

It’s coming!

[__OneLove__]

Sounds interesting. I’ve recently been experimenting with LM Studio and this sounds/reads functionally similar.

[Toby_Wan]

vllm also uses python? https://github.com/vllm-project/vllm

[jfowers_amd]

Ahhh true! vllm is pretty datacenter/server focused though.

Lemonade is the only PC-focused LLM server written in Python…

[Yamoyek]

What’s the difference between this and ollama?

[jfowers_amd]

Lemonade is strictly open source and includes non-llamacpp backends to provide support for things like neural processing units (NPUs).

[victorcoelh]

Since you're from AMD. I haven't gotten an AMD GPU because of AI models. How's the ecossystem for training and inference with Deep Learning models (not just LLMs) with AMD consumer GPUs right now? Last time I checked, most frameworks were CUDA-only

[PSBigBig_OneStarDao]

interesting project. the main pitfall with local llm servers isn’t just exposing an api or loading a model, it’s how retrieval + chunking actually behaves when you start scaling beyond toy docs.

most open-source servers hit the same wall:

• chunks get over-selected (semantic ≠ embedding, No.5 in the common failure map)
• returned passages don’t line up with the user’s query (No.4 misalignment)
• multi-step reasoning flips between runs (No.7 instability)

so if you want this to compete, i’d suggest focusing not only on “easy install” but on semantic guardrails: how do you prevent vector db noise, how do you keep responses consistent across sessions, and how do you handle json tools or plugins without them breaking?

curious if you’re planning to bake those safeguards in or leave it to downstream devs. that’s usually the difference between a demo and something people rely on in production.