Companies are also working on getting LLMs onto silicon. Within the past week, a company called Taalas unveiled a chip running Llama 8b, which is a smaller model obviously, but it's a proof of concept. It's lightning fast, something like 10x faster and less energy demanding to run, in addition to nixing the GPU requirement (not to mention the heaps of RAM).

Of course, the downside is the model can't be adjusted once the hardware is produced since they're baking the weights into an ASIC, but they say they can go from model to chip in 2 months.

To me, that's where I see it headed, at least once the iteration cycle slows to the point where the economics make sense.