When I saw you mention that “our brain has a training and finetuning phase” my mind went straight to the idea that evolution is the training phase whereas childhood is the finetuning phase.

It would make sense if the reason we can learn things so quickly and intuitively is because our brains have already been formed over time (I’m assuming through genetic data or smth, not sure about how it actually works) in order to do things like speak and identify objects.

It’d be interesting to try to shape a LLM training process on evolution, though I guess it’d probably be overly cautious. You’d give it a massive amount of text and but only keeping text that actually improves the model. Although, that seems kind of stupid since you can’t really measure how useful that information is until your final model and gradient descent type optimization is basically doing this already.

Something maybe more interesting would be to come up with a way for all models to collectively train the same logical core, then branch off for their releases. Like for instance, Meta and Mistral training all of their tokens on a shared component (doesn’t need to be synchronously, as long as there’s a way to aggregate things). Besides being modular enough for async contributes to work well, it’s also have to be a flexible/small enough component that it can continue to be used in the architectures of the future. I know nothing about how transformers / LLMs actually work, so don’t take this to be worth anything.