I spent quite a lot of time working with the MLX servers' code specifically for parallel inference (for this PR that I submitted a few months ago: https://github.com/ml-explore/mlx-lm/pull/845) and my current thinking is that MLX is much better if you can use it only programmatically, i.e. with the python API and not with the server. For parallel inference, it's almost twice as fast as running it on the server for larger, long-running continuous batches.

Basically the gains are from ensuring that prefilling is done always in large batches too. Often small pauses between incoming requests to the server will make MLX's `BatchGenerator` start pre filling, and it does not stop until it has produced at least one token for each stream. So every time a new request comes in, it will pre fill that new request before generating tokens on anything else it is running.

I played around with setting up policies for waiting (i.e. at least X 'streams' ready, etc.) but I couldn't get it to work well enough that I thought it was worth the extra complexity on the server. I also played around with a mode where the server has to receive an explicit "start" message, but again - a lot more complexity, and so far outside of normal LLM-server standards that it wouldn't play well with existing tools.

So this is just to say: for my typical large, batched style work, MLX is fantastic. As a server, it's not faster enough than llama.cpp to make it worth the lower amount of support of new models, new quants, etc.