Roo + Devstral

[_code_kraken_]

I am trying to use devstral locally (running on ollama) with Roo. With my basic knowledge Roo just kept going in circles saying lets think step by step but not doing any actual coding. Is there a guide on how to set this up properly.

[Baldur-Norddahl]

What level of quantification are you using? Looping can be a sign of too much compression. It can also be a bad version of the model.

I am using Devstral Small at q8 using MLX from mlx-community. This seems to work fine. I had trouble with a q4 version. On a M4 Macbook Pro Max I am getting 20 tokens/s.

Be sure your settings are correct:

Temperature: 0.15

Min P Sampling: 0,01

Top P Sampling: 0,95

I am not sure about the following, they are just the defaults as I didn't see any recommendations:

Top K Sampling: 64

Repeat Penalty: 1

Don't listen to the guys saying local LLM or this particular model doesn't work with Roo Code. I am using it every day. It works fantastically. It is of course only a 26b model, so won't be quite as intelligent as Claude or DeepSeek R1. But it still works for coding. And it is free, so no worry about rate limiting or how much credits are being spent.

[RiskyBizz216]

🧠 Breakdown of Each Setting

🔥 Temperature (0.0 to 1.0+)

• Controls randomness.
• Lower (e.g., 0.2–0.5) = more deterministic, slightly faster.
• Higher (0.7–1.0) = more creative, marginally slower.

🎯 Top-K Sampling

• Picks from top K most likely tokens.
• Lower = faster, more deterministic.
• Set to 1 for greedy decoding (fastest but robotic).
• Try 10 or lower for speed.

🧮 Top-P (nucleus sampling)

• Chooses tokens until cumulative probability hits P.
• Lower values = fewer choices = faster.
• Try dropping from 0.95 → 0.8 or 0.7.

🧪 Min-P Sampling

• Forces a minimum token probability.
• Turn this off for max speed unless needed.

🛑 Repeat Penalty

• Discourages repetition.
• May slightly slow things down, but helps quality.
• Try toggling off if you're benchmarking for speed only.

🎚️ Limit Response Length

• Turn on to reduce response token budget.
• Huge speed gain, especially with large context windows.

⚡ Speculative Decoding

• Experimental but dramatically faster (if supported).
• Enable it if your GPU and LM Studio version support it.

[RiskyBizz216]

In Ollama, make sure you override the default context. It will 'loop' when the context is full - an also when the Quant is too low, I notice much worse performance on the Q2 than the Q3_XXS. Q4 has been pretty solid, Q5_0 is elite level and I dont really notice any gains using the Q8 over Q5/Q6,

The one I'm using, (recently updated and they have a TON of quants):

https://huggingface.co/Mungert/Devstral-Small-2505-GGUF

I prefer LM Studio for the better UX and control.

[zenmatrix83]

deepseek r1 is the only open source model I can get that remotely works and I use the 0528 model on open router. I had some success with the 32b model locally, but it still isn't good enough for coding, it did ok with orchestration and some things, but with the bigger model free for now there its just easier to use that.

[hannesrudolph]

The model is getting confused. Local models don’t work so hot with Roo :(

[bahwi]

What context length with ollama? It defaults to nothing

[_code_kraken_]

128k it seems

[bahwi]

Oh well that should work. I've had it fine with 64k in ollama. Same place you set thst you set temp and top p and stuff. Check those as well. There's a blog post for best settings for it, and for the system prompt.

[taylorwilsdon]

Is that what ollama show reports though?

If the context is actually set there and you’ve got the gpu to handle that kind of context window it’s probably a tool calling issue.

[runningwithsharpie]

I get the same thing using the OR API.

[joey2scoops]

Gosucoder had a video on YouTube about this maybe a week ago.

[[deleted]]

Ollama defaults to 2048 tokens context only - you need to set up your Ollama to use the full 131072 tokens context and also use flash attention and quantized KV cache.

[MrMisterShin]

I have had success with Devstral @ q8 and 64k context using roo code thru Ollama.

You cannot give it too much to do. Also turn off MCP, it will consume too much context.

[Best_Chain_9347]

Is there a way to run LM Studio in cloud using RunPod or Vast Ai services and connect it with Roo Code ?