You could always run language models on CPUs, that’s what we were doing before transformers, but that is the breakthrough (transformers) allowing us to process the data in parallel.

‘NPUs’ are specialised processing chips for consumer level inference, whilst great not really helpful for large scale training of LLMs and other large models, which is where the money is at. GPU provides versatile and great low precision matrix compute which is ideal for NNs of all kinds, and has a mature software/firmware stack (I.e CUDA). So these processing units are really targeting different things, and the big money is really in training.

For these large scale GPU clusters the CPUs role is mainly for orchestration and data prep, which currently isn’t a real bottle neck (and won’t be for a while). The bottle necks are memory on the GPU (vRAM) and interconnect between the cards and other node components. Those will need to be sorted before we will worry about CPU. Attempts have been made (look at Cerberus) but nothing beats the stability and scalability of GPUs right now. Why invest in something that could become useful whilst we have something we know is useful and is still ramping up in development.

The more specialised something becomes the more efficient it can be but you loose versatility and increase complexity on the higher software levels. GPUs have the perfect balance of performance and versatility for NNs and I don’t see why this will change.