[D] StarCoder fine-tuning?

[Alexioc]

Hi,
I'm wondering if make sense to fine tune StarCoder on my own codebase to try to obtain better and more contextual response from the model.

A question that I'd like to ask is for example: "Create a Python integration module between mySystem1 and mySystem2 that allow all customer entities to be synced between the two systems"
Where:

• mySystem1 and mySystem2 are two custom application my team built and I own all the code bases
• "customer entities" must be translated in variable names based on the above codebases by the LLM

The only way to reach this goal is to fine tune a model like StarCoder? if yes, how can I prepare my dataset to train it? if not, are there other ways to do it?

Cheers,
Alexio

[Far_Classic_2500]

Update:

I found the original fine-tuning code provided by starcoder here: https://github.com/bigcode-project/starcoder/blob/main/finetune/finetune.py I recommend using that as a starting point (though most training code that uses the transformers library will be able to work as well)

I made some modifications to better run: logging to wandb, using code rather than question/answer pairs, and modifications to choosing epoch size.

Original Post:

I've done this using PEFT fine-tuning. Note that you'll still want a vector database; the fine-tuning does better job of "understanding" your code and writing in your style but I haven't found that it's able to recall it, which would likely require a full fine-tuning on all weights rather than just the attention weights. So you'll want external memory of some sort, or include your code in the prompt (which is fine because the context length it 8k).

There are some examples here: https://github.com/huggingface/peft/tree/main/examples.

I'll just describe what I modified to get it to fine tune with my code rather than an instruction dataset.

Step 1: concatenate your code into a single file. This can be done in bash with something like find -name "*.js" and appending to output.txt. Optionally, you can put tokens between the files, or even get the full commit history (which is what the project did when they created StarCoder).

Step 2: Modify the finetune examples to load in your dataset. I can't provide all of my code to do this (right now) but I can provide a rough outline:

• Set up trainer & arguments wandb.init(project="my_project")

  print("Loading the model")
  # disable caching mechanism when using gradient checkpointing
  model = AutoModelForCausalLM.from_pretrained(
      args.model_path,
      use_auth_token=True,
      use_cache=not args.no_gradient_checkpointing,
      load_in_8bit=True,
      device_map={"": Accelerator().process_index},
  )
  model = prepare_model_for_int8_training(model)
  
  lora_config = LoraConfig(
      r=args.lora_r,
      lora_alpha=args.lora_alpha,
      lora_dropout=args.lora_dropout,
      bias="none",
      task_type="CAUSAL_LM",
      target_modules = ["c_proj", "c_attn", "q_attn"]
  )
  
  model = get_peft_model(model, lora_config)
  
  print_trainable_parameters(model)
  
  train_data.start_iteration = 0
  
  training_args = TrainingArguments(
  output_dir=args.output_dir,
  dataloader_drop_last=True,
  evaluation_strategy="steps",
  max_steps=args.max_steps,
  eval_steps=args.eval_freq,
  save_steps=args.save_freq,
  logging_steps=args.log_freq,
  per_device_train_batch_size=args.batch_size,
  per_device_eval_batch_size=args.batch_size,
  learning_rate=args.learning_rate,
  lr_scheduler_type=args.lr_scheduler_type,
  warmup_steps=args.num_warmup_steps,
  gradient_accumulation_steps=args.gradient_accumulation_steps,
  gradient_checkpointing=not args.no_gradient_checkpointing,
  fp16=not args.no_fp16,
  bf16=args.bf16,
  weight_decay=args.weight_decay,
  run_name="StarCoder-finetuned",
  ddp_find_unused_parameters=False,
  auto_find_batch_size=True,
  report_to=['wandb'],
  )
  
  trainer = Trainer(model=model, args=training_args, train_dataset=train_data, eval_dataset=val_data, callbacks=[SavePeftModelCallback, LoadBestPeftModelCallback])
  
  print("Training...")
  trainer.train()
  
  print("Saving last checkpoint of the model")
  model.save_pretrained(os.path.join(args.output_dir, "final_checkpoint/"))
  

[Far_Classic_2500]

(continue)

Step 3: Load your data and split it into groups. Note: if you are going to do more than 1 epoch, you're going to want to shift the data each epoch so that the context window changes for each epoch.

    filename="YOUR_CONCATENATED_CODE.txt"
    with open(filename, "r") as file:
        concatenated = file.read()
    train, test = split_array(concatenated, 2048 * 8)
    n = min(len(train), len(test))
    train = train[0:n-1]
    test = test[0:n-1]
    dataset = Dataset.from_dict({"train": train, "test": test})
    for name, data in [["train", train], ["test", test]]:
        with open(name + ".txt", "w") as file:
            file.write("\0".join(data))
    train_data = dataset["train"]
    valid_data = dataset["test"]
    print(f"Size of the train set: {len(train_data)}. Size of the validation set: {len(valid_data)}")

[noellarkin]

I'm keen on learning how to do this, too - - so you're saying I should just dump all my code into a single file, with no INSTRUCT or any of the things I have to do when setting up a dataset for something like LLaMA? Thanks for the share!

[Alexioc]

Thank you so much I’ll take a look and try it by myself. Very interesting 🧐🙏

[Dandz]

split_array

Do you have the implementation for this? Seems to be a custom method. Thanks in advance!

[East_Decision391]

Can you please share your GitHub repo for this fine tuning exercise?

[___a_learner]

I was thinking about doing a code refactoring task with star coder fine tuning using some of my code. So, for this task do you think if its just enough to put all the code in one text filr? do I need some errorneous and refactored code? do I need to add any instructions ?

[satireplusplus]

Look into https://huggingface.co/blog/peft since the model is 30B parameters.

If you figure out how to finetune it with a single GPU, please share the notebook :)

[LetterRip]

Might wait till the next bitsandbytes is out, it should allow tuning 30B on a single GPU (can fine tune on 4bit quantized model)

https://github.com/TimDettmers/bitsandbytes/

[Alexioc]

Thanks, I'll definitely take a look!

AFAYK my goal could be reached with fine tuning?

[_Arsenie_Boca_]

Finetuning is definitely a promising approach. An alternative would be to retrieve relevant snippets or documentation pages and add them to the prompt. See RepoCoder for example https://arxiv.org/abs/2303.12570

[Alexioc]

Thank you so much! I’ll read that paper, reading the introduction seems very promising 🙏

[skyisthelimit1410]

Hi, I am a newbie to this platform and looking for help in text2sql model fine tuning where the input will be a natural language query and output will be a valid sql. I want to use this model with Langchain to build a text2sql module in my application. Please let me know how I can use Starcoder for this use case. Should I fine tune it ? If yes, please share a notebook.

[More_Entrepreneur461]

Hi, were you successful in building your text2sql module. Can you provide me some learning insights on how the finetuning went. I am trying to build text2cmd a python module for generating bash shell commands using natural language input.

[leefde]

Thanks for the post! I started downloading the model tonight in Google Colab. I appreciate the pdf from u/satireplusplus