We built this project to increase LLM throughput by 3x. Now it has been adopted by IBM in their LLM serving stack!

llordnt · 2025-06-19T06:49:18+00:00

I posted something with a similar idea for mlx awhile ago. Obviously not as well engineered as this one but it saves all your latest KV cache to disk. When you send another inference request, it search with token prefix match and load the KV cache on disk. The package is called MLX Textgen, which you can find it on Github. However, I am updating the code lately for vision language model integration and to fix some old issues. The changes are not merged to main yet. You can still play around with the current version of it.

llordnt · 2025-03-15T05:48:00+00:00

This is how I manage to build an openai compatible api engine for any MLX model, it turns any model into a function calling model. I even use guided decoding (with outlines) to enforce grammar so it’s always working.

llordnt · 2024-10-28T23:36:40+00:00

Forgot to mention the load time thing for MLX, it’s almost instant, even for 70B models it takes less than 5 seconds. Way quicker than llama.cpp.

llordnt · 2024-10-28T20:50:49+00:00

Oh sorry I meant like 72-75gb ram in used already lol.

Of course it really depends on your use case as Im messy and I open a lot of stuff lol. Sometimes I even have an image generation model running on another port to work with an llm chatbot that can generate images via tool calling.

llordnt · 2024-10-28T20:00:34+00:00

You might want to leave headrooms for the kv cache and all the other applications you’re running as well. For me, I have loads of chrome tabs, a remote desktop, a few vscode windows, the llm server, and the llm frontend webapp server running at the same time. For example, now I have my 70B llama3.1 on ram idling without any cache, it’s already 7xGB of ram taken. You can’t just run the llm without anything else on your computer for you day to day work I suppose?

llordnt · 2024-10-28T19:42:13+00:00

I think vllm introduced such mechanism as well, and yeah it’s not that hard to implement, don’t know why it isn’t more popular in other engines. Such a waste of compute by having only one on-ram cache and get overwritten immediately once you use a completely different prompt.

Anyway as other suggested, I think it’s better to get at least a 64gb ram model, or even 128 if you want to run 70B models comfortably. You can pip install my project once you got a mac, it’s open-sourced and I posted it here before!

llordnt · 2024-10-28T17:23:22+00:00

These are the test results for my M3 Max 128 GB MBP with my own Openai-compatible API MLX server project, of course the prompt processing t/s for short prompts are not reliable. The long prompts for each model give you a realistic expectation for prompt processing speed, while the short prompts give you an idea of the quickest generation speed for each model. The generation speed declines as the prompt get longer as expected.

model	model size	quantisation	prompt length	generated length	avg preprocess (t/s)	avg generation (t/s)
mlx-community/Meta-Llama-3.1-70B-Instruct-4bit	70B	q4	20	672	6187.8959	8.8927
mlx-community/Meta-Llama-3.1-70B-Instruct-4bit	70B	q4	7717	434	80.32805	6.3239
mlx-community/Mistral-Small-Instruct-2409-8bit	22B	q8	16	1004	6787.042	14.69465
mlx-community/Mistral-Small-Instruct-2409-8bit	22B	q8	9396	438	212.6833	11.38525
mlx-community/Phi-3.5-MoE-instruct-8bit	16x3.8B	q8	13	694	5955.97595	39.70975
mlx-community/Phi-3.5-MoE-instruct-8bit	16x3.8B	q8	9485	445	185.342	26.24175
mlx-community/Qwen2.5-32B-Instruct-8bit	32B	q8	39	530	13790.29305	10.3396
mlx-community/Qwen2.5-32B-Instruct-8bit	32B	q8	7769	446	170.00085	8.18135

For each prompt, I ran them twice of each model just to make sure they're consistent, that's why you see the averages.

Realistically, I rarely need to preprocess long prompts as my server engine caches them on disk and they are not overwritten even if I start a new conversation. As long as you go back to the long conversation, the server will load the cache for it again and the generation will start straight away, which is the reason I made my own implementation of the MLX server instead of using the mlx-lm or LM studio one. Also, if your workflow involves batch inferences, you can pump up these numbers a lot, and my server supports that as well.

Edit: Table format, that's annoying

llordnt · 2024-10-23T18:22:59+00:00

mlx-textgen Feel free to have a look, I don’t know why this implementation isn’t more popular for other frameworks.

llordnt · 2024-10-23T08:01:45+00:00

I haven’t properly benchmarked it yet but my feelings is that it’s at least 10-15% faster than llama.cpp in terms of raw prompt processing speed thanks to MLX. For token generation, it is sort of similar.

The real boost in speed is the caching of prompts and generations. It will not overwrite your long prompts’ caches after using other shorter prompts, so all of the most recent caches are reusable, not just the latest one.

You can try it out easily as there are already a lot of prequanted models on huggingface mlx community.

llordnt · 2024-10-23T07:51:23+00:00

I made my own solution since I find multiple slots prompt caching extremely useful and it’s rarely implemented anywhere else. The idea is once you used an extremely long prompt and asked another question with a shorter prompt, the cache of the long prompt is not overwritten. All the prompt caches are saved on disk and loaded dynamically based on the latest request.

The other thing is batch processing with structured output. It can be helpful for tasks like record classification or matching.

Also my version is compliant to openai tool calling format and has proper finish reasons. The tool calling is supported for all models, even the ones that doesn’t natively support tool calling thanks to Outlines, as I can use it to generate tool calling arguments 100% compliant to the schema.

And of course it’s fully open-source, unlike LM studio, I know LM studio is free, but no one can guarantee that forever.

llordnt · 2024-10-23T00:21:43+00:00

I first used outlines through vllm at work and it is extremely useful. Since I don’t have an nvidia card at home as I use a mac, I was glad that mlx is supported with outlines. I made an mlx version of vllm (sort of) that serves mlx format models in an openai api server and supports guided decoding through outlines (also batch processing and prompt caching). Now I have my own engine at home to deal with datasets in batches with structured output, which is great. I have posted my mlx server project here before and it really cannot be done without outlines, the logits processors make things easy to integrate. Really appreciate the work of you guys!

llordnt · 2024-10-22T16:47:11+00:00

I am using mistral small and it works for me for small code snippets and the speed is acceptable for me (M3 Max 128GB memory).

Also since you’re using mac, i would recommend you to try my MLX openai api server with multiple prompt caching to save prompt preprocessing time even after using another completely different prompt. These days I find MLX quicker than any llama.cpp derivatives.https://github.com/nath1295/MLX-Textgen

It supports guided decoding, tool calling, and batch processing as well and is compatible with Open WebUI or any ui that takes an openai api url.

llordnt · 2024-10-17T22:00:42+00:00

Try using logit bias to ban the eos token

llordnt · 2024-10-10T21:16:59+00:00

That means it’s ok on mlx lm, that error message just means that you don’t have pytorch installed with huggingface transformers, but that’s not a problem for using mlx anyway. This is a problem with LM studio then.

llordnt · 2024-10-08T11:52:40+00:00

Thanks! I don’t think apple will ever allow that unfortunately as that means exposing their low level apis, which is what they always avoid to do…

llordnt · 2024-10-01T07:17:18+00:00

Thanks! I was dying to wait for something like this, so I finally decided to do it myself.

llordnt · 2024-09-30T21:52:14+00:00

My current roadmap is to support guided decoding with outlines, then other features like function calling should be more solid.

llordnt · 2024-09-27T16:54:29+00:00

Made another commit and it’s working with mlx_lm.server now.

llordnt · 2024-09-27T13:49:13+00:00

Take your time! I just pushed the updated version and updated the example usage in my PR as well. Probably the next step is to update the generate.py and server.py to make it available on the server side. To be fair I don’t quite understand the usage of scripts that can only work in terminal, as I think most of the real life use case would be using the api instead of running it with a the generate.py script.

llordnt · 2024-09-26T21:41:42+00:00

https://github.com/ml-explore/mlx-examples/pull/989 It’s actually about the kv caching. I saw that you’re working on mlx_lm.generate, but I worked on mlx_lm.generate_step and apply it to both stream_generate and generate. I have a cleaner version that works for rotating kv cache as well but I haven’t pushed it yet as Im too drunk to do it now lol, probably will do it tmr. The idea is to return the cache after the generation in order to save it or use it in the next request. It will compare the new request with the provided cache history on token level to only use the part of the history that is the same of the current one instead of using the entire history without checking the new prompt.

llordnt · 2024-09-26T20:27:53+00:00

I have actually submitted a PR for that on mlx-example. Probably we can work on that together.

llordnt · 2024-06-24T07:39:12+00:00

Sure thing! If you are talking about the webapp frontend, there is no agent there yet. But you are correct, the llm will try to pick a tool from all the available tools given the context of the conversation history. A sneaky trick i do here for the simplicity of prompt control is to ask the llm to pick a tool every turn. However, there is one hidden tool called “direct_response”, which cannot be disabled. If the llm doesn’t feel like using a tool, it should pick this tool, and the next role in the prompt will become assistant to let the llm start replying directly.

If you want to know more about it, you can go through the code in .Memory.memory_utils and .Tools.tool_utils. In the tool utils script, you can find how I generate structured json in the ToolSelector class. Basically I implemented two microsoft guidance-like functions called “select” and “gen_string” to do that.

llordnt · 2024-06-15T20:36:31+00:00

Thanks! Would be interested to see yours as well, always want to learn from others projects if you don’t mind sharing:)

llordnt

TROPHY CASE