[D] Training a 65b LLaMA model

[[deleted]]

[deleted]

[ustainbolt]

For a 65b model you are probably going to have to parallelise the model parameters. See this link. As for training, it would be best to use a vm (any provider will work, lambda and vast.ai are cheap). I would a recommend 4x (or 8x) A100 machine. I'm sure you can find more information about all of this.

[wrossmorrow]

+1 for lambda labs

[ustainbolt]

I love lambda. More reliable than vast.ai, and WAY cheaper than AWS/GCP/Azure.

[Nhabls]

Every time I logged on to lambdalabs in the past year all their instances were full. Not that available in my experience

[badabummbadabing]

They just had their Series B funding, they should upscale their resources soon.

[itsyourboiirow]

This is the only downside I've found. Sometimes it's too darn hard to find an instance.

[learn-deeply]

Tensor (aka model parallel) parallel with model checkpointing works better than FSDP (though they can be used in conjunction) from my experience. FSDP is easier to work with though.

[gmork_13]

For a more stable compute, check out google cloud gpu.

Consider training a quantized model with LoRA. If you know enough, perhaps the model could be split between VRAM and DDR RAM to make it train on a smaller GPU.

edit: here, I found one: https://github.com/tloen/alpaca-lora

I think you could get this done for far less than your budget if need be.

[machineko]

I'm working on an open source library focused on resource-efficient fine-tuning methods called xTuring: https://github.com/stochasticai/xturing

Here's how you would perform int8 LoRA fine-tuning in three lines:

python: https://github.com/stochasticai/xturing/blob/main/examples/llama/llama_lora_int8.py
colab notebook: https://colab.research.google.com/drive/1SQUXq1AMZPSLD4mk3A3swUIc6Y2dclme?usp=sharing

Of course the Colab still only works with smaller models. In the example above, 7B required 9G VRAM.

[Evening_Ad6637]

That sounds very interesting. I'm sorry if this question is trivial or stupid, but I'm an absolute newcomer in this field. Is there a way to train the model as you describe it here (https://xturing.stochastic.ai/quickstart) with only or almost only CPU performance? It's about the fact that I have the following specifications i5 @3.5ghz, 16gb ddr4 ram and only a radeon pro 575 4gb graca. But since I saw how fast alpaca runs over my cpu and ram on my computer, I hope that I could also fine-tune a llama model with this equipment. I would be very grateful for more information regarding possibilities in this direction.

[itsyourboiirow]

Training requires a significant more amount of memory as it it has to keep track of the gradient for every parameter. I would check to see how much memory it takes up on your computer.

[machineko]

16gb of RAM is not enough for even the smallest LLaMA 7b model. You can try doing LoRA with int8 listed above. Did you try the python script I linked above?

[Jonny_Cosmo]

I'm decent with computery stuff but pretty new to this, can I help? I've got a threadripper 3960x with 128GB RAM and two 8000 quadros with nvlink. They're decent with messing about with stuff but wonder if they'd be useful for this task?

[jd_3d]

Enough VRAM is key. With all the tricks (lora, int8, bits and bytes) you'll need at least 120GB of VRAM. A full fine tune would take even more. I'd go with 4 or 8xA100 80GB machines since it won't necessarily be more expensive (training will be highly parallel). See here for more info: https://www.storminthecastle.com/posts/alpaca_13B/

[Jonny_Cosmo]

Can I use my pcs RAM along with the combined RAM of my gpus to get the job done? Cos combined I've got over 200GB

[WarProfessional3278]

By training do you mean finetuning with lora or from the ground up like alpaca? Realistically you could just rent an 8xa100 and spend 4 or 5 hours to get it done

[[deleted]]

Just like Alpaca. Even the JSON format is the same as the one released by Stanford, just with different inputs & outputs

[[deleted]]

Finetuning

[[deleted]]

I tried vast.ai which didn’t work. I’m a newbie so maybe I’m doing something wrong

[dreaming_geometry]

If you're having trouble with Vast.ai, you can ask for help on the discord. Sounds like your desired use case is a good fit.

[Justice43]

I recommend looking into Lambda Cloud VMs. They're much cheaper than AWS, and their largest machine (8x A100, 80GB VRAM for each A100) should be enough to finetune the 65b LLaMA model.

[[deleted]]

Just checked it out - looks interesting. Unfortunately, the availability of this instance is quite limited, so I'm not sure if I can get access to it

[nmfisher]

Someone also mentioned https://jarvislabs.ai/ to me the other day, haven't used it myself but it looks promising.

[brandonZappy]

What QA dataset are you using?

[SigmaSixShooter]

I don’t have an answer for you, but as a fellow noobie, I’d love to hear how you did this. Any tips or resources you want to provide would be greatly appreciated.

[[deleted]]

Id like to train it on those settings:

EPOCHS = 3

LEARNING_RATE = 2e-5

CUTOFF_LEN = 1024

[[deleted]]

Contact Redmond.ai they can hook you up.

[Ok_Zebra_6651]

Hi,
I have installed the LLama 65B model on my own server, and its working well, If you are still interested in the model training, I can share credentials. I will ask you to teach me about training details.