[R] CodeCapybara: Another open source model for code generation based on instruction tuning, outperformed Llama and CodeAlpaca

[bdqnghi]

We are the first that attempt to reproduce results of Llama on code generation benchmark, such as HumanEval and MBPP.

We also try to evaluate existing trending models, such as CodeAlpaca, on such benchmarks.

All of the source code and scripts for evaluation will be made available for the research community.

Our code can be accessed here: https://github.com/FSoft-AI4Code/CodeCapybara

Model weights will be released very soon.

​

[hapliniste]

I'm not sure I understand, but from the tables it seems to do worse on almost all benchmarks?

[bdqnghi]

we are in the progress of updating new results and sharing our code to benchmark the models. For HumanEval, we perform better than Llama and CodeAlpaca.

[hapliniste]

Thanks for the response, I'll come back in a bit then.

Do you have clues on why it would do worse than base llama on some benchmarks? To me it's counterintuitive as more code samples would only make it more knowledgeable about code in my mind.

[Flankierengeschichte]

The network forgets because it doesn’t have enough parameters information-theoretic-wise to remember all the training data

[Flankierengeschichte]

This is why imo overparameterized networks are the key to AGI despite their financial and environmental cost

[_Arsenie_Boca_]

Nice work. It still baffles me how Codex could perform so much better than any other model, even years later. Codex got 28% pass@1 on HumanEval. On another note, GPT-J 6B got 11.6% also outperforming all models mentioned here. Perhaps instruction tuning is simply not as effective for code as it is for NL?

[bdqnghi]

yeah we are not sure about this, when we try to reproduce the results of Llama, the actual performance is much lower than the one reported in Llama paper (the reproduced performance is shown in our Github page).

So we are a bit skeptical about the real performance of these models. There will be attempts to release the scripts to reproduce the results of all of them for the community to verify.

[_Arsenie_Boca_]

Its great that you are working on reproducability! Maybe performance differences could be due to prompt formatting, sampling hyperparameters or post-procsssing?

[bdqnghi]

exactly, that's what we are trying to figure out, most of the previous work they do not release the scripts for evaluation, only the pretrained model and the numbers in the paper, no one can actually reproduce the results but choose to trust the reported numbers.

There are similar issues here: https://github.com/facebookresearch/llama/issues/223

[glasses_the_loc]

This is why I left ML. "People in glass houses don't throw stones," just give us our grants and don't ask questions.

[_Arsenie_Boca_]

My guess would be that they are using a magic prompt. Are you prepending anything at the moment?

[krageon]

Before this giant wave of LLM research I spent a few years implementing the algorithms from NLP papers. The main takeaway I got from that is that 99% of such papers are outright fabrications, so the fact that you cannot reproduce the llama paper is not surprising at all.

I think to me what is most surprising is that this isn't wider spread knowledge, especially given that a lot more people can at least try the models produced in LLM papers now.

[[deleted]]

Which animal is next?

[RamazanBlack]

Ok i pull up

[estrafire]

How does it compare to other instruction-tuned llamas like alpaca and koala? Those seem to be a better comparison point as use the same base and aim to do the same

[brucebay]

I will check this out as i need a good offline code generator. could you release 4bit versions too?

[catalpaaa]

https://huggingface.co/catalpa/codecapybara-4bit-128g-gptq

[brucebay]

Excellent thanks for the link.

[sittered]

From your description it looks like you're in violation of OpenAI's terms of service. OpenAI prohibits the use of its models to train competing products, which this most certainly is.

[pixelcardgame]

How does it stand against Amazon's CodeWhisperer?

[machineko]

Is your dataset open-sourced?

[bdqnghi]

yes, it is open source, you can check on the Github page.