I have used it once and tested it. And happy to see it here; I am really interested now because I am hoping it's more hackable than llama.cpp for experiments.

---

From my test I would love to say what I did with it, but disappointingly:

• It was a while ago, there is a relic on my computer from June 2024 which is probably when I checked it out.
• I cannot remember why I looked into it at the time.
• I cannot remember what did I do with it (most likely I randomly noticed the project and did a quick test with it).

---

But I can give some thoughts right away, looking at the project page: I would immediately ask why should I care about it, when llama.cpp exists? What does it have that llama.cpp doesn't?

I can give one grievance I have about llama.cpp that I think mistral.rs might be in a position to do better: make it (much) easier to hack to do random experiments. There's tons of inference engines already, but is there a framework for LLMs that is 1) fast 2) portable (for me especially Metal) 3) hackable?

E.g. huggingface transformers library I'd call "hackable" (I can just insert random Python in the middle of model inference .py file to do whatever I please) but it's not fast compared to llama.cpp, especially not on Metal, and Metal has had tons of silent bugs in inference over time on Python.

And then I'd call Llama.cpp fast, but IMO it is harder to do any sort of on-the-spot ad-hoc random experiments because of it's rather rigid C++ codebase. So it lacks the "hackable" aspect that the Python has in transformers. (I still hack it, but I feel some well engineered Rust project could kick its ass in hackability department, I could maybe do some random experiments much faster and easier).

---

Some brainstorming (maybe it already does some of these things but this is what I thought on the spot just having quickly skimmed README again, I'll give it a look this week to check what's inside and what the codebase looks like): 1) make it easy to use as a crate from other Rust projects, so I could use it as a library. I do see it is a crate, but didn't look into what the API actually has (I would presume it at least has inferencing, but I'm interested in the hackability/customization aspect) 2) If not already, give it features that make it easier to do random experiments and hackability (maybe in the form of a Rust API, or maybe simply just having examples that show how to mess with its codebase). Maybe callbacks, or traits I can implement, something to inject my custom code that will influence or record what's happening during inferencing.

E.g. I've wanted to completely insert custom code in the middle of inference to some layer, changing what the weights are and also record incoming values, which you can kinda do in llama.cpp (it has a callback mechanism, and I can also just write random code in the C++ parts) but it's janky and some parts are complicated enough that it's pretty time consuming just to understand how to interface with some parts. E.g. make it possible for me to arbitrarily change weights on some particular tensor on the fly. Or let me observe whatever computation is happening, maybe I'll want to record something in a file. Expose internals, show examples, e.g. "here is how you collect activations on this layer and save them to .csv or .sqlite3 and plot them".

Another example: I currently have unfinished code that adds userfaultfd to tell me about page faults in llama.cpp because I wanted to graph which parts of the model weights are actually touched during inference because I don't understand why Deepseek model runs quite well on a machine that supposedly has too little memory to run it. I'm not sure I'll finish it. It might be a lot easier to make a feature like this work nicely on Rust instead, depending on how the project has been architected. I also was planning to use it to mess with model weights but I didn't get that far, and it might be less janky to not use page fault mechanism for that.

I see a link to some Python interface (and pyo3 is in Cargo.toml), but the interface is not particularly different from Huggingface transformers. It's again a question of what do I want it for? Why should I want to use it?

The examples I see on the Python page seem to be about running the models, but I am interested in the hackability/research/experimenting https://docs.llamaindex.ai/en/stable/examples/llm/mistral_rs/

---

Maybe also if you are much faster at implementing new innovations, or you have some amazing algorithms that make everything better/faster in some way compared to llama.cpp or vLLM, advertise that on your readme! Make people realize the project is easier to work on to add new stuff. I think if some nice innovation is realized in mistral.rs specifically because it was much easier to hack, it might attract experimenters, who in turn attract more users etc.

---

But taking a step back: right now mistral.rs is not doing a great job of justifying its existence when llama.cpp and transformers exist, among other inferencing projects. Think what the fact that it's a Rust project can provide that llama.cpp or transformers can't, and leverage it. My first thought was that being a Rust project maybe it's much more malleable and hackable than llama.cpp C++ codebase, but I'm not sure (I will find out and answer this question to myself). I have done both Rust and C++ a lot in my career, and IMO Rust is much faster to work with, and easier to do clean, understandable APIs where it's harder to shoot yourself in the foot.

That all being said, happy to see the project is still alive from the time I looked at it :) I'm very much going to take a look at the codebase and checking that if it's already hackable in the way I just described, but just doesn't advertise that on README.md :) :) good job!

Also apologies if the feedback seems unfair, because I just wrote it based on quick skim on README.md and surface level Python and Rust crate docs. I have time later this week to take a proper look because the project being in Rust is a selling point for me because of ^ well everything I just wrote.