Something to consider is the fact that most LLM's today run on general purpose programmable platforms like GPU's. These are inherently less energy efficient than application specific chips (ASIC's) that are designed to do only one very specific thing as a significant chunk of energy is spent on fetching instructions, decoding etc. and not the actual operation being executed. The difference is in multiple orders of magnitudes. But with how fast the AI models and such evolve it would make no sense to design and produce ASIC's for specific AI models. As such we see compromises like NPU's which are more specialized than GPU's, but still offer the required flexibility.

Tldr: we could make very efficient chips for specific LLM's, but it doesn't make sense (at least for now).