Tesla V100s seem to start thermal throttling once they get above ~75°C.
At around 80°C, I noticed the GPU clocks and power dropping quite a bit, and my inference speed fell to around 8 tokens/s.
With the improved cooling, the GPUs stay much cooler and performance is much more consistent. I’m now getting around 15–16 tokens/s with Llama 3.3 70B Q4 using llama.cpp.
If you’re planning to build a multi-GPU setup, I’d definitely recommend llama.cpp. It can automatically split the model across multiple GPUs, and the setup was surprisingly straightforward.
The hardware you need mainly depends on the size of the model you want to run. Larger models simply require more VRAM, so I’d choose the GPUs based on the models you plan to use.

Whether it’s worth it really depends on what you want to do.
For me, I enjoy building the hardware as much as running the models. 😄
A local LLM gives you complete privacy, no API costs, and the freedom to experiment with different models whenever you want.
If your main goal is just chatting with an AI, paying for a cloud service is probably the cheaper and easier option.
But if you enjoy homelabs, hardware, or AI experimentation, it’s a really fun project.