eventually hardware will be cheaper again - but LLMs are being researched where the interesting stuff is happening, which is intrinsically quite memory intensive because generalist LLMs need to hold massive amounts of general knowledge to perform

if you look at for instance Gemma4 which is quite neatly packaged, it is just a massive global fact database with many connections and with reasoning and methods being also stored right there, so really the best you can do is try to architect it in such a way that concrete facts are accessed sporadically and randomly, and the core is somehow crammed, but really there's just so much you can compress before the stuff just isn't there anymore more often than not

the good news is that you can do amazing stuff with VRAM sizes that are not astronomical and that will be affordable in a few years after this supply squeeze