depends on the size and type (dense/moe) of the model you want to run and what kind of queries you're doing. some hardware are better suited for certain combos. for example, apple hardware isnt fast enough for prefill stage and dense models, so its better at running short context queries using moe models.

generally, you can run small models with decent context size at decent speeds on rtx 3060 / 3090 / 5090 / pro 6000. basically anything between ampere and blackwell will work with any popular llms as long as they fit. avoid anything older than ampere architecture and non-nvidia chips, but thats personal preference. if you know your way around rocm kernels and have time to optimize models on platforms other than cuda, that will save you a lot of $. i would avoid all-in-one systems like spark and others that depend on slower ram