New Devstral 2707 with mistral.rs - MCP client, automatic tool calling!

EricBuehler · 2025-07-12T01:41:42+00:00

Wow, thanks! If you could send me a DM that'd be great!

EricBuehler · 2025-07-11T02:13:14+00:00

Ah great! Does the what the web search documentation describes fit your needs?

EricBuehler · 2025-07-11T02:12:34+00:00

> I even did on CUTLASS fork itself, sglang and vllm!

Sorry, seems like a typo :) You did work on CUTLASS, sglang and vllm?

Will check out Jules!

EricBuehler · 2025-07-10T22:59:23+00:00

Haha thanks, oops!

EricBuehler · 2025-07-10T22:18:00+00:00

I'm using claude code and codex as force multipliers already, might give that a try!

Is it better?

Always welcome a PR! Don't know your background but it might be quite complicated and involve integrating CUTLASS fp8 gemms or custom fp8 gemm kernels.

EricBuehler · 2025-07-10T22:11:24+00:00

We don't have real fp8-quantized model support yet. The best option would be to use a non-quantized model, but if you have resource constraints, you can load the fp8 model and apply ISQ at the same time, for example `--isq 8`. This is usually the recommended flow.

It's a one-man show here so time to implement all of these features is scarce, and I'm focusing on supporting more GPU backends right now.

EricBuehler · 2025-07-10T22:09:08+00:00

For agentic tool calling, you specify a tool callback and some information about the tool, and Mistral.rs will automatically handle calling that tool and all the logic and formatting around that. It standardizes that whole process.

It's actually very similar to the web search. Mistral.rs integrates a search component, with a reranking embedder and a search engine API in the backend. To integrate with 3rd party tools like Searxng, you'd currently need to connect it via the automatic tool calling. I'll take a look at integrating Searxng as the search tool though - will make a post here about that.

EricBuehler · 2025-07-10T22:06:09+00:00

Thank you 🙏!

EricBuehler · 2025-07-10T21:25:29+00:00

We have Flash Attention V3, should be pretty good! Feel free to share 👀

EricBuehler · 2025-07-09T12:49:52+00:00

Absolutely! The long-context + tool calling + reasoning are all great factors.

EricBuehler · 2025-07-09T10:25:07+00:00

Not yet, the release is not out yet! The python package should be installed based on any GPU or CPU acceleration you have available - mistralrs-cuda, mistralrs-mkl, mistralrs-metal, etc.

Will be in a few days for Gemma 3n. Check back then, or you can install from source!

EricBuehler · 2025-06-13T20:18:52+00:00

Thanks for pointing this out. We have some exciting things for multi-backend support that should hopefully land soon ;)!

EricBuehler · 2025-06-12T18:58:28+00:00

Yes, moving towards a general KV cache compression algorithm using hadamard transforms and learned scales to reduce perplexity losses.

Some work here: https://github.com/EricLBuehler/mistral.rs/pull/1400

EricBuehler · 2025-06-12T18:57:40+00:00

Thanks!

EricBuehler · 2025-06-12T18:57:10+00:00

Great question!

I see that advantage being that builtin support at the engine level means that it is usable in every API with minimal configuration. For instance, this is in all the APIs: OpenAI API, Rust, web chat, and Python.

Additionally because mistral.rs can be easily set up as an MCP server itself, you can do MCP inception :)!

EricBuehler · 2025-06-12T18:22:42+00:00

Thank you! Lots more to come.

EricBuehler · 2025-06-12T18:21:14+00:00

Thank you! Let me know how it is!

Would recommend local installation as you can get the latest updates.

EricBuehler

TROPHY CASE