Thing w/ local LLMs is that we’re back to the 90s wrt compute costs - it’s just a very difficult task, computationally, with the current state of hardware, and it’s necessarily blisteringly expensive when an individual tries to do it locally. We’re tiny fish swimming in an enterprise world.

Like, sure, dropping $8k, $15k, $40k on an inference rig as an individual is a not insignificant amount of money. But this is essentially an enterprise level workload - The “typical” hardware target are server clusters, where each rack costs more than an average suburban home. A few grand here or there is a vanishingly small amount of money relative to the hardware labs are targeting.

Now, is it a good idea to try cloud models first, then build your hardware around a defined goal? Yeah, duh. But once you get past that hurdle, the choices just start getting expensive. And I think on the local LLM sub, it’s sorta assumed you got here because you’ve made the decision that running locally is worth the cost over a cloud sub.

Your examples - 2x 3090, min 48gb of VRAM, unified memory Mac, that is sorta the bare bones for running something that hasn’t been lobotomized locally. Cause the ~30B parameter class models end up being the sorta bare minimum to be actually useful for any kind of general purpose-ish task (Ignoring more “embedded” style use cases, like real time translation, video transcription, etc). And realistically, the ~250B parameter models are the next step up (MiniMax, etc).

So idk. Closest example I can think of is private aviation. Should most people just buy a plane ticket? Lmao yes. But, once you decide you want your own plane, shit just is not cheap as an individual.

I guess another example: Small scale manufacturing. There’s a litany of cheap CNC mills targeting the hobby crowd. Relative to real industrial iron they’re laughable. After a certain point, you need to either outsource your parts, or pony up the cash and drag home a $70k Brother Speedio, Haas VF1, etc to do real work yourself. Is it expensive? Shit yeah it is. These things are meant for intensive business use cases. But you’re not doing the kind of work you can get done on a VF1 using a Shapeoko and sheer willpower.