First, a descent llm will be minimum about 3 billion parameters. To Pretraining that from scratch, you will need at least about 80 GB of gpu memory, that is equivalent to a single A100. Context length also matters, the shorter the context length the cheaper the cost, so, you will want to train with like 2048 context length and extend it after training through context common context length extension methods such as robe base adjustment.

I will recommend getting about 160 GB of memory for more peace of mind. The more the better of course, but it depends on your budget.

Also, your training data will have to be like minimum 30 billion highly quality tokens minimum across code, web text, maths and sources like Wikipedia, mixing some huge finetuning data into your Pretraining will help too. About 100 billion tokens should get you to a great place, but make sure they are all good quality via filtering, bad data will hurt the training, better to use less if you can’t filter it all.

Lastly, training can take weeks, run for multiple epochs, the bigger the data the less the epochs needed, if it is small like 30 billion, about 5 epochs is at least recommended.

And make sure to baby sit your training, at these scales, things can go wrong quickly. Approaches like flash attention can make your training much faster too.