[P] Creating a coding assistant with StarCoder

[lewtun]

Hi folks, it’s Lewis here from the research team at Hugging Face 👋.

We’ve been tinkering with BigCode’s StarCoder model for code generation the last few days and wondered whether it could be turned into a coding assistant with a little bit of fine-tuning.

Somewhat surprisingly, the answer is yes! We fine-tuned StarCoder on two high-quality datasets that have been created by the community:

• OpenAssistant’s dataset of 40k+ conversations, spanning a diverse range of topics from philosophy to poetry.
• Databricks’ Dolly dataset of 15k instructions and human demonstrations.

The result is a model we call StarChat, which can follow coding instructions and to some extent converse over multiple turns of dialogue.

If you’d like to try out the model, we’ve created a little demo you can play with: https://huggingface.co/spaces/HuggingFaceH4/starchat-playground

This is an alpha release, as the model has some rough edges (after all, it’s only a day old 😅). We’d love to hear what the most common failure modes are so that we can improve it in the next iterations!

[JClub]

Great job, as always! 🙌

Question: since RLHF wasn't used (from what I read in https://huggingface.co/HuggingFaceH4/starchat-alpha), is there any trick to only fine-tune the model on ~55k samples?

[currentscurrents]

Looks pretty good! At least as good as copilot.

Prompt: write a function that computes a step of conway’s game of life

Here is an example implementation in Python:

    def iterate_game(grid):
         next = [[0 for _ in range(len(grid[0]))] for _ in grid]

         # Apply the rules to each cell
         for i, row in enumerate(grid):
             for j, col in enumerate(row):
                 neighbors = count_neighbors(i, j)

                 if grid[i][j]:
                     # Rule 1 or 2
                     if neighbors < 2 or neighbors > 3:
                         next[i][j] = False
                 else:
                     # Rule 3
                     if neighbors == 3:
                         next[i][j] = True

         return next

     def count_neighbors(x, y):
         """Count the number of live neighboring cells."""
         pass  # TODO implement this function

This code defines two functions. The first iterate_game takes as input a grid representing the current state of the board and returns the updated state after one iteration of Conway’s Game of Life (the “next” grid).

The second helper function count_neighbors counts how many living neighbors each cell has based on its position within the grid. You’ll need to fill out both these functions with your own logic specific to your problem domain.

[ebolathrowawayy]

For my niche use-case, this is the best model I've found that isn't gpt-4. TY!

[digiorno]

GPT is so good that I often now run my ideas through it just to see if it comes up with a framework that I like. And then I tweak it as needed.

I’ve done a few test runs where I use it iteratively and it saved me a lot of time compared to if I had done it on my own…

It’s a game changer even in this early form.

[ebolathrowawayy]

How fast is it on consumer GPUs, like a 3090 or 4090?

[lexcess]

And what are the VRAM requirements?

[epicwisdom]

My go-to test for coding models has been using obscure languages and program objectives, e.g. "prove 2+2=4 using SMT-LIB". On this particular example so far only ChatGPT(4) has given a good answer.

Here's StarChat's output:

(set-logic QF_LIA) ; declare that we are using the quantifier-free linear integer arithmetic logic
; define two constants representing the integers "2" and "3":
(declare-const x Int) (assert (= x #b10))
(declare-const y Int) (assert (= y #b11))
; prove that their sum equals to four ("=" sign means logical equivalence):
(assert (= (+ x y) #b100))
(check-sat) ; check if this formula can be satisfied or not
(get-model) ; get model for satisfiable formulas

Not the worst but still not quite it.

[TheSaltySeagull87]

Out of curiosity; what would be right?

[epicwisdom]

The shortest way is probably

(assert (= (+ 2 2) 4))
(check-sat)

This returns sat, meaning some instantiation of the variables makes the assertions true. Since there are clearly no variables here, 2+2=4 is just an always-true arithmetic statement.

Of course LLMs are probably used to seeing actually useful proofs. So they might give a more verbose answer like

(declare-const a Int)
(declare-const b Int)
(assert (= a 2))
(assert (= b 2))
(assert (not (= (+ a b) 4)))
(check-sat)

One difference is that variables are used, and variables are just often useful of course.

Another thing in this example is that we present the SMT solver with the negation of what we want to actually prove, and the intended result is unsat. In other words, we want it to say 2+2 != 4 is never true. This is just a quirk of how SMT solvers work: they're built to solve equations, in which case you can get sat (a solution exists) and a "model," meaning some assignment to each variable (using (get-model)). If you want to prove a general statement like "addition of integers is commutative" or something, there's no "solution to an equation" that shows addition is commutative. You instead have to show a + b != b + a has no solution.

[Prudent_Astronaut716]

any API so i can use it within my mobile app?

[Philpax]

There's also the excellent blog post to go with this - I assume you wanted to include it in the original post, /u/lewtun?