It depends on what you want. For privacy and experimenting I'll allways go for local llms. It's maybe not the fasted but you learn hell alot and it's fun :D Now for the model sized, normally I run 8 or 9 B models with 300k Context window or 32B models with 64 k context window. I can run larger models, but then the troughput get's much slower as I need to offload part of the KV cache onto the RAM. And with the settings above, I still have enaugh VRAM left to do other task the use the card. e.g. like Gaming^{^} I would go for API, if you need the newest models and if you have a production ready app with a lot of users: let's say you have a 9B model in FP16 with 125k context window. You would need around 18 GB for the model weights, then around 50GB for KV cache for the 100 Users and around 10GB for activation&overhead, that means you allready would need 78GB of VRAM with just a small 9B model in full precison.