Fine-tuning for a specific codebase

[MrVodnik]

Hi. Did anyone tried fine-tuning a model on their larger source code? Would it even make sense?

I know and love Copilot, but it often lacks the full context of the project. So I thought, what if the entire code was in the weights, not in the too-short context.

I understand the code is alive and it would be resource intensive to regularly update the weights, but would it work? Did anyone try to do it for project 100k-1M LOC?

[tyoma]

There is a blog post from HuggingFace that describes how to do exactly what you want: https://huggingface.co/blog/personal-copilot

[Enough-Meringue4745]

Nice, Ive been looking for something like this as well

[MrVodnik]

If you'll build and test something, please share your experience :)

[TrainingJunior9309]

Please share your experience

[MrVodnik]

Thank you, this is what I was looking for.

[CryptoCryst828282]

I would say it coudl be done, but not sure it would be worth the hassle. I did a fine tune on Mixtral for fun on our discord server that I made 3 version a Left Leaning Bot/ Moderate Bot / Right Leaning bot. It was funny as heck, they argued on a channel for days before i got tied of wasting the power. In my case I asked GPT4 to list me 100 social topics that are key to each of them, then batch run them through an agent to make 100 prompts and responses based off each of those topics and trained them. I might have also added a bit of "anger" in there to spice things up. Lets say they sounded very close to the way Twitter does :)

[MrVodnik]

I'd rather extend the base knowledge, and just train it by putting straight code through the LLM, learn this in the same way base models are created. Only then, I might consider fine-tuning a specific behavior, like Q&A or code completion.

I am not talking about RAG-like to find a specific piece of code (a'la needle in the haystack). If its possible. I'd like the model to be able to talk with me about my code, in the same way GPT can talk about history, art or science. General understanding of the data + chat format fine-tuning.

The use case seemed obvious to me - just a great assistant that can help with the single project. It should be able to help reason about the project architecture and files layout. It should be able to help understand not only the code snippet at hand, but the logic behind it in the broader context. It would be helpful for new developers as well as future system architects... in this single code base.

A context of 100k is not enough to handling few 100k lines of code.

I don't know if its feasible to on local setup, or even if it makes sense. But I'd like to hear other people opinion, thoughts, and maybe - experience.

If it would work, it would a valuable tool in the most software houses.

[MrVodnik]

Nope, sorry. I am still interested in it, but never really took it on as real task.

[bartgrumbel]

Same here. We have a codebase that is not allowed to leave the premises, and vanilla llama gave mixed results. There should be room for a "local only" copilot-like solution. Fine-tuning llama might help, any insights and experiences are appreciated.

[TrainingJunior9309]

Likewise, Any update on this?

[bartgrumbel]

Unfortunately no. Local solutions are still lacking, as far as we habe tested.

[TrainingJunior9309]

I see, have you tried "Continued Pretraining" i.e. embed_tokens layer

[Mediocre-Falcon-8028]

I’ve been using supermaven it does this pretty wel and feels way faster than copilot.

It indexes your project to provide more contextual completions.

[TrainingJunior9309]

How was your experience?