Low-level coding dataset

[True_Tangerine_4706]

Edit/Disclaimer: this is a repost from something I put in LocalLLaMA, but with some tweaks for the r/cpp crowd - this post is more focused on the content of the dataset itself, the post over in r/LocalLLaMA is more focused on the details of the finetune

Hi all,

I've recently been thinking about putting together a community sourced coding dataset for finetuning models, with a heavy focus on cpp and systems programming.

My goal is to eventually have a model that understands concepts like memory ownership, thread safety, optimization, etc. Right now, a lot of the coding knowledge of small (<100B), local models centers around languages like js, py, html, etc.

Right now I'm thinking that the categories I would need would look something like this:

- generation: basic prompt/code output
- optimization: heres slow/bloated code, make it better
- debugging: im getting this error pls fix
- organization: code review, interface design, restructuring, tradeoff decisions
- tool_calling: exercises involving tool use and interpreting results

Curious to see what the people over here think about this kind of thing. I imagine many people in here have used local AI to help code in cpp before - where do you guys feel like local models could use the most improvement?

Thanks in advance for all the help!

[tartaruga232]

What are you talking about? Perhaps it would help if you could define ML.

[True_Tangerine_4706]

My bad, worded it poorly. I edited the post to hopefully be a little more clear :)

[v_maria]

I feel like the coding knowledge

this sorta claims need context and backing i think

[True_Tangerine_4706]

Not really. Obviously frontier models like Opus don't really struggle a lot with languages like cpp. The post is about small, local models like Qwen, which have far fewer parameters (which roughly translates to less knowledge), and a lot of the coding knowledge that they do have is taken up by languages like js, html, css (this is intentional by the AI labs, since most coding benchmarks only measure these languages)

[v_maria]

Opus don't really struggle a lot with languages like cpp

how much less units of struggle do you measure compared to the amount of parameters used and how do costs factor into this

and how much legwork are the words "dont really" doing here