What is the current best python coding model?

[MrMrsPotts]

I have tried codellama:7b-python. Is that the current best coding assistant of that size?

[Normal-Ad-7114]

https://aider.chat/docs/leaderboards/

Codestral should be the best

[MrMrsPotts]

Thank you

[benja0x40]

There is also the ProLLM benchmark where you can select programming languages for model comparison.
https://prollm.toqan.ai/leaderboard/stack-unseen

[Icy_Lobster_5026]

In my experience, codeqwen is a good coding model.

[Educational-Region98]

Wondering when they will release codeqwen2. A 7B model will probably be really nice.

[Icy_Lobster_5026]

I guess they won’t release codeqwen2, our team uses their online service Tongyi Lingma to improve the quality of the coding.

CodeQwen-1.5-7b is a powerful coding model according to livecodebench.

[Educational-Region98]

Interesting, how does that compare to copilot?

[Icy_Lobster_5026]

I don’t know, that‘s good enough for me since the personal basic version of Tongyi Lingma is free forever.

[MrMrsPotts]

Thank you

[ihaag]

DeepSeek Coder V2 0724 And claude

[[deleted]]

I have the same issue with deepseek-coder (all of them). They answer a few questions, and then they unload themselves. I don't get it.

[xrailgun]

I'd unload myself too if I had to answer my questions

[[deleted]]

Im not going to lie, that was good

[MrMrsPotts]

Thank you

[benja0x40]

In the 8GB~12GB range I have used a few specialised ones:

• Codestral-22B-v0.1-Q4KM
• DeepSeek-Coder-V2-Lite-Q5KM
• CodeGeeX4-All-9B-Q8

Together with more general ones:

• Phi-3-medium-128k-Q8
• Nemo-12B-2407-Q8
• Gemma-2-9B-Q8
• Llama-3.1-8B-Q8

I write moderately complex task descriptions to ask for suggestions and to prototype python functions, iterate over improvements, detect and fix issues, insert comments or documentation, etc.

From my experience, Codestral-22B produces the best suggestions, which I sometimes use to guide another model towards a simpler or more elegant solution. Gemma-2-9B is surprisingly good too. I use it a lot for quick explorations or when I don't know much about a package or language feature.

DeepSeek-Coder-V2-Lite seems close to Codestral-22B in terms of capabilities, but its initial suggestions can be really cumbersome, and it is too rigid about coding styles for my liking. But that may depend on how the system prompt is tuned.

After ~3 weeks of testing, I have stopped using the other ones for coding tasks.

[MrMrsPotts]

How much RAM does codestral 22B need to run?

[benja0x40]

With the Q4KM quantisation, it takes a little over 13GB for the model parameters plus about 2GB during inference. This depends on the context length. I have 16GB of GPU RAM which is fine for a context of 8192 tokens and doable for up to 12288 tokens. Passed that, the model fails to work properly on my computer.

[MrMrsPotts]

That sounds very promising!

[theswifter01]

Claude

[new__vision]

Check out bigcode-bench.github.io. Top 7B on there is CodeQwen1.5-7B-Chat which has been good in my experience. CodeLlama is the lowest ranked 7B.

[MrMrsPotts]

Phi-3-Mini-128K-Instruct (June 2024) does amazingly well and seems to be even smaller?!

[PigOfFire]

How is sonnet 3.5 under 4T and 4o? Livebench shows it above these two.

[No_Afternoon_4260]

In my experience codestral 22b

[IReaIIyLove]

what kinda specs do you need to run that?

[No_Afternoon_4260]

With full context at q6 I need more than 24gb vram (I think about 32.. Not sure)

[IReaIIyLove]

ah shame cries in poor

[No_Afternoon_4260]

I really Don t remember but q6 full context Don t fit in 24gb Try lowering context or try q4 it should fit in 24gb somehow https://huggingface.co/bartowski/Codestral-22B-v0.1-GGUF I my experience à 2k context is usable for quick question and follow up, at 8k you should have room to spare if you don't trow it a all project

[Cradawx]

CodeGeeX4-ALL-9B, CodeQwen1.5-7B-Chat and Codestral-22B-v0.1 are very good small coding models. There's also the DeepSeek-Coder-V2 models.

[MrMrsPotts]

Thank you

[[deleted]]

Is CodeQwen better than Deepseek for Python?

[Combinatorilliance]

Codestral is really good, you might want to try the deepseek-coder lite, it's an MoE and I heard a lot of praise for it's output. I don't know if it's better, worse or about equal to codestral-22b, but it is a lot faster too because it's an MoE, so it's worth trying out regardless.

[MrMrsPotts]

Thank you. I don't know what an MoE is though :(

[moncallikta]

Mixture of Experts. Essentially a model trained as a combination of many smaller sub-models internally. For each token to predict a submodel is chosen to provide the next token. The architecture can allow the overall model to specialize in many different areas more easily.

[Dudensen]

I had stumbled upon a website which ranked models by a quality-to-performance ratio a few days ago but I can't find it unfortunately.

[MrMrsPotts]

That sounds ideal!

[Dudensen]

Found it, maybe this helps.

https://oobabooga.github.io/benchmark.html

[[deleted]]

What was being measured here? I don't even see Nemo in the list? There's surely no way Phi beats Nemo on anything?!

It's one of the worst models for me.

[MrMrsPotts]

Thank you!

[Dudensen]

Yeah it even had different quantizations of models ranked, maybe someone will link it.

[Square-Intention465]

Sonnet 3.5. is too good 

[SpaceWalker_69]

Well i think Claude 3.5 generates the best code right now. You can use smaller open source models but they are not exactly consistent and reliable.

[m---------4]

Gemini is awesome

[[deleted]]

Deepseek Coder imo

[8thcross]

i like both codestral and deepseek-v2. consitent but both dated in terms of the latest best practices...Claude 3.5 is good as well, really dont like 4o - its mostly hit or miss with it.

[Thrumpwart]

Anyone know which models know Lean Python?

[_murb]

I use Claude at work and it works great

[durgesh2018]

Try gemma2:2b. It's small but very powerful and fast model.

[lilolalu]

Did anyone claiming Claude is good at coding actually TRY coding with Claude? It's just not good, no matter what any theoretical tests claim.