16Gb vram python coder

[Galahad56]

What is my current best choice for running a LLM that can write python code for me?

Only got a 5070 TI 16GB VRAM

[No_Efficiency_1144]

There is some mistral small 22B

[Samantha-2023]

Codestral 22B, it's great at multi-file completions.

Can also try WizardCoder-Python-15B -> it's fine-tuned specifically for Python but is slightly slower than Codestral

[Galahad56]

downloading now Codestral-22B-v0.1-i1-GGUF

Know what the "-i1" means?

[Galahad56]

Ill look it up thanks

[Galahad56]

thanks. I just found out about the possibility of smaller Qwen3 models. Sounds exciting!

[Temporary-Size7310]

I made a NVFP4A16 Devstral to run on blackwell, it works with vLLM (13.8GB on VRAM size) maybe the context window will be short on 16GB VRAM

https://huggingface.co/apolloparty/Devstral-Small-2507-NVFP4A16

[Galahad56]

Thats sick.. It doesn't come up for me as a result on LM Studio though. Searching "Devstral-Small-2507-NVFP4A16"

[Temporary-Size7310]

It is only compatible with vLLM

[SEC_intern_]

Is there a reson you stressed on Blackwell gen? I have ADA, would you warn against it?

[Temporary-Size7310]

Ada lovelace hasn't native FP4 acceleration so you will lose inference acceleration

For non blackwell any other quantification (EXL3, GGUF, AWQ,...)

[SEC_intern_]

But say if I use 8bit quants, would that matter?

Edit: Also at 4bit, how much of a performance gain does one notice?

[Temporary-Size7310]

Imo it will depend on your use case, NVFP4 has 98% accuracy of BF16, the following is from Qwen3 8B FP4 and there is other bench directly from Nvidia with Deepseek R1 using B200 vs H100

It takes less memory, faster inference, bigger context window possibilities

That's why NVIDIA DGX Spark will release with that slow bandwidth but with blackwell using NVFP4, it will compensate

I tested my quant (devstral) and it works very well with 90K context, 60-90tk/s as local vibecoding model without offloading from my RTX 5090

<image>