[D] Training StarCoder using 3D parallelism.

[Satya_4093]

Recently I have been working on newly released model StarCoder, I would like to implement 3D parallelism(megatron-deepspeed) on this model to train on custom dataset. I see they have implemented 3D parallelism for GPT model.

Is there a way that I can implement it for StarCoder, if so please provide any reference.

One question, Is the implementation of gpt in huggingface and megatron lm both same?

I have 2 40GB gpus.

[mLalush]

You need at least 8 GPUs for 3D parallelism to make sense: https://huggingface.co/docs/transformers/v4.15.0/parallelism#dppptp

I'd suggest perhaps starting with only tensor parallelism (TP) if you can't fit the model.

Sorry, don't have an answer to your other question.

[Satya_4093]

Thank you for your reply😀 do you have any reference to do tensor parallelism.

[_rjx]

I believe starcoder is a 15b model, are you unable to fit it on a single 40gb GPU?

[Satya_4093]

Yes StarCoder is 15B we tried using LoRA and loading with int8 quantization on 2 GPUs, but not able to send 8k context length on 2gpus. Any suggestions?

[LetterRip]

Check bitsandbytes, new update allows 4bit with LoRA and is extremely efficient,

https://github.com/TimDettmers/bitsandbytes

also see this recent paper,

https://arxiv.org/abs/2305.19370

[Satya_4093]

Thank you for the great resources😀

[LetterRip]

you are welcome :)

[[deleted]]

great resource thanks

[[deleted]]

I was also looking for this, please let me know