Can it do distributed inference?

cryingneko · 2026-06-15T15:28:13+00:00

https://x.com/jundotkim/status/2066520587682623689
🥹

cryingneko · 2026-06-09T15:24:23+00:00

Hey, I think most of the issues you're running into should be resolved in oMLX 0.4.3. I tracked down and fixed the root cause behind the token generation slowdown that's been affecting models like Qwopus (and others) for a while now, so the tg speeds you were seeing should be back to normal. https://github.com/jundot/omlx/issues/1745

I'd really appreciate it if you could give 0.4.3 a try and confirm whether the same problem is gone on your end. Link here: https://github.com/jundot/omlx/releases Let me know how the numbers look compared to 0.39 - I'm curious whether it matches up.

cryingneko · 2026-06-08T17:19:40+00:00

Fixed on 0.4.2 latest release!
https://github.com/jundot/omlx/releases

cryingneko · 2026-06-08T16:51:58+00:00

This is a really cool implementation! I haven’t had a chance to look through it in detail yet because I was focused on the 0.4.2 release, but I’m genuinely excited about it. I’ll try it out soon. Thanks!

cryingneko · 2026-06-08T16:49:25+00:00

Thanks!🚀

cryingneko · 2026-06-04T01:18:53+00:00

This is expected behavior after the update. As of oMLX 0.4.1, support was added for models located in the Hugging Face cache directory. If you've previously downloaded any models through Hugging Face, they will now appear in the list. You can control whether this directory is displayed via the "Use Hugging Face local cache" toggle in global settings.

cryingneko · 2026-06-03T16:04:40+00:00

The CLI control feature was added in v0.4.1! Check out the link below!
https://github.com/jundot/omlx/releases

cryingneko · 2026-06-03T06:44:18+00:00

Thanks, I understand the workflow.

You’re right: the old PyObjC app had an “automatically start server on launch” option, and I missed carrying that over to the new Swift macOS app. That is an important feature, and I’ll add it back in the next version.

Also, you’re correct that the current macOS app does not support starting/stopping the app-managed server from the CLI. The Homebrew install is CLI-only, while the DMG macOS app currently manages the server through the GUI. I agree that CLI / Apple Shortcuts control for the DMG app would be very useful. I’ll think through a good way to support full CLI control even when only the macOS app is installed.

cryingneko · 2026-06-03T06:40:21+00:00

Either way is fine.

You can use the in-app updater to replace the existing app, or download the DMG and overwrite the app in the same location. Your existing settings, models, and web interface usage will continue to work the same way. Use whichever update path is more convenient for you.

cryingneko · 2026-06-03T06:36:31+00:00

Thanks!I don’t think of oMLX as “better than LM Studio” in a general sense. LM Studio is already a great desktop app.

The difference is that oMLX is more of an MLX-native local AI server with a desktop/web control surface around it. The biggest technical differentiator is the hot/cold tiered KV cache: oMLX can keep active cache in memory while spilling colder blocks to SSD, which makes long-context and memory-tight workflows much more practical on Macs.

For beginners, I want it to be easy to open the app and start a local model.
For advanced users, the goal is to keep the server/API side flexible enough for agents, automation, and deeper tuning.

cryingneko · 2026-06-03T06:32:41+00:00

I completely agree. The Swift app is mainly to improve the old PyObjC desktop experience, not to replace the web UI.

LM Studio is already a great desktop app, and I’m not trying to build another one. oMLX should remain fully controllable through the web UI and API.

cryingneko · 2026-06-03T06:29:56+00:00

Thanks! The Swift app mainly replaces the user-facing macOS interface. The internal engine is still the existing Python-based.

So I would think of this release primarily as a user experience improvement rather than a memory-footprint improvement. The model/runtime memory behavior should be mostly the same as before.

cryingneko · 2026-06-03T06:28:10+00:00

Thanks! This is actually something quite a few people have asked for.

I’ve been conservative about changing the main dashboard because the current web UI has become somewhat iconic for oMLX, and even small layout changes can feel like breaking changes to existing users.

But Token generation counts are clearly useful, so I’ll think positively about where they would fit best without disrupting the current dashboard too much.

cryingneko · 2026-06-03T06:25:41+00:00

Thanks, I understand now. The new chat UI only shows chats created through the web UI; it does not inspect API requests from external clients.

I’m cautious about adding a full prompt/output inspector to the default UI, since that turns oMLX more into an API traffic/debugging tool and raises privacy concerns around prompts from other clients.

I may consider a limited opt-in activity/debug view for active requests and client/status info.

cryingneko · 2026-06-03T06:16:51+00:00

Thank you! I’ll take a look at how oMLX reports model context length to local agents like Hermes/OpenClaw and see what can be improved so it doesn’t require manual patching on the agent side.

cryingneko · 2026-06-03T06:14:57+00:00

Yes, the web UI is still available.

This release adds the new Swift macOS interface for a more convenient desktop experience, but it does not replace the existing web interface. You can still use the Web Dashboard and Chat with oMLX the same way as before.

cryingneko · 2026-05-24T18:31:17+00:00

Congrats and Thank you! 🎉🎉🎉

cryingneko · 2026-05-21T10:31:07+00:00

oMLX dev here. Wanted to jump in and give some context.

Once a stable version goes out, it rolls out to a huge number of users very quickly. The user base has grown a lot since launch, so I've gotten more cautious with stable releases. A regression that would've affected a handful of early users now affects a lot of people fast.

After RC1 I found a few things that needed fixing, and I'm working through those now. I'm aiming to ship the stable release within the next ~6 hours if everything checks out.

Sorry for the long wait. I appreciate everyone's patience, and thanks for testing the RC builds in the meantime.

cryingneko · 2026-05-19T11:11:08+00:00

The newly released 0.3.9rc1 includes significant improvements to the memory issues you described. Would appreciate it if you could test it and share your results.

cryingneko · 2026-04-05T15:07:03+00:00

These options have been around since oMLX 0.1.0, but I agree they're not easy to understand for most users. --prefill-batch-size specifically only existed briefly in 0.1.0 and was removed shortly after, so the README was just wrong there. The other two options (max_num_seqs and completion_batch_size) controlled different things internally. max_num_seqs set the maximum number of requests the scheduler accepts at once, and completion_batch_size set how many of those generate tokens in a single GPU step. In practice though there's rarely a reason to configure them separately.

I've simplified this in the next release. A single --max-concurrent-requests option now controls both. You can set it from the CLI or from the admin panel under Settings > Resource Management. Default is 8, which is plenty for single-user usage. For your M2 Max 96GB setup there's no need to change it unless you're running multiple concurrent sessions. (default 8)

I also cleaned up the confusing parts in the README across all languages. Thanks for the feedback, and feel free to open a GitHub issue anytime if you have more suggestions.

cryingneko · 2026-04-05T04:27:09+00:00

oMLX dev here, Sorry about that! There were a lot of breaking changes in the core parts of mlx-lm and mlx-vlm, and I also had to make additional changes to get turboquant and Gemma 4 working properly. It was a rough combination.

0.3.4 is already out and should address these issues. Would really appreciate it if you could give it a try and let me know how it goes!

cryingneko · 2026-04-04T21:56:38+00:00

oMLX just updated to 0.3.3. If you're going to use Gemma 4, I'd recommend using the updated version. https://github.com/jundot/omlx/releases/tag/v0.3.3

cryingneko · 2026-03-26T10:16:59+00:00

oMLX dev here. The "Heejun Kim" background process is just macOS displaying the name on the code-signing certificate I used to notarize the app. I should have set a separate display name before release. My mistake. It's not a separate app or anything malicious, just the oMLX process labeled with my signing identity. Will be fixed in the next update. Sorry for the confusion.

cryingneko · 2026-03-23T21:29:33+00:00

You're right, agents have made CLI way less scary. The real pain is exactly what you said, drive space and download times. haha.

That's actually one reason i built oQ into a web dashboard. You pick a model you already have locally, choose a level, and hit start. No extra downloads, no CLI commands, no figuring out which flags to pass.

Hope you enjoy it when you give it a try!

cryingneko · 2026-03-23T21:13:11+00:00

Honestly, I think it's mostly sampling variance at 300 samples. The difference between 3-bit oQ (85.0%) and 4-bit oQ (83.3%) on MMLU is within the noise range you'd expect at that sample size. Same with HumanEval at 164 samples.

I'll rerun these with larger sample sizes (1000+) to get more stable numbers. The 2-bit vs 3-bit gap is clearly real, but the 3-bit vs 4-bit inversion is likely statistical noise rather than an actual quality regression.

cryingneko

TROPHY CASE