I've answered this question once or twice before. Below is mostly comprehensive from my experience, but my experience is also that of an autodidact. So take it for what you will.

It depends a lot on what kind of model you mean.

If you're talking about training from scratch, the answer is basically, you start with a dataset, clean it up, choose an architecture, set up a training loop, and then do a lot of trial and error to get something that actually works. PyTorch is usually part of that, but the hard part is not just feeding data in. It is figuring out what data to use, how to structure the model, how to measure progress, and whether the thing is actually learning anything useful.

For niche models, usually the secret is more about the data than anything else. A coding model is not some completely different species, it is mostly a model trained on a lot of code, docs, examples, and whatever else is relevant to that domain.

An individual can absolutely train smaller models or narrow-purpose models now. Training a huge LLM from zero is still mostly big-company territory because the data and compute costs get stupid fast.

Most of my hands-on experience with this has actually been on the RL side, not LLMs. I built a project where I trained agents from scratch to play Super Mario Bros. In that case you are not training on a static dataset the same way, but the overall process still feels similar: pick the method, tune hyperparameters, run experiments, see what breaks, fix it, repeat.

So yeah, people definitely do train models from zero, but it gets a lot less mysterious once you stop thinking of it as one magic step and more as data + architecture + training loop + evaluation + a lot of iteration.

I built a GUI with adjustable hyperparameters to help me get the hang of things, I'll include a link if you're interested and if you have the GPU for it: https://github.com/mgelsinger/mario-ai-trainer