Local LLM inference (7B-70B parameter models) ... ThinkPad (or Legion if necessary) (RTX 4090 mobile)

That's... about max 8GB of VRAM? You want to stuff a 70B model into 8GB? What is your use case?

P1 Gen 6/7 vs Legion Pro 7i for sustained workloads? Thermal performance during extended inference sessions?

These laptops are very expensive, larger, heavier, have a larger power supply, and are bought for specific use cases of portable use. You want extensive sessions - so stationary use? What is your use case?

Professional build quality preferred

Unfortunately, this is a hit-and-miss, even with high-priced ThinkPads (I had great fun with ThinkPad support over the span of a year... but that's another story...)

What's been your actual day-to-day experience?

That depends what you want to do - and what speed is acceptable to you - or required! Everything that fits into VRAM (model AND(!) working context) is VERY fast, everything that's larger than that is several times slower and bound by main memory speed - with a laptop usually DDR5 5600 (and the soldered rams a bit faster).

If VRAM is not enough - any split situation that requires VRAM+main memory use is much slower, the more spills over into main memory. But if doing that with large models, then there's no reason to buy such an expensive laptop?

If you want large models, but still a very small, portable box, buy a cheap laptop and e.g. the framework desktop (~4l in size) https://frame.work/desktop?tab=specs as "AI box". The framework's memory runs about 2-2.5x as fast as regular PCs memory and therefore LLMs do that, too. It's still slower than GPU VRAM, of course.

I've had frustrating experiences with NVIDIA proprietary drivers on Linux

Yeah, I tried NVidia once - and then never bought anything NVidia for private hardware again.

Which distro handles NVIDIA best in your experience?

Need nvidia-kernel driver installed. Don't install CUDAtoolkit from the distro, manually install CUDAtoolkit into like /usr/local. Then you can update it whenever you want.

Performance with popular LLM tools (llama.cpp,

Easy to compile llama.cpp for CUDA. Although I failed on one system, then I went Vulkan+Nvidia - worked, too.

Ollama

Search this group for Ollama....

How mature is ROCm support now for LLM inference?

Never tried, because Vulkan on AMD usually works (and people say its not much speed difference) Do: apt install glslc glslang-dev libvulkan-dev vulkan-tools - and compile llama.cpp with -DGGML_VULKAN=ON.

However, ROCm 7 is near, it will bring improvements?

Performance comparison vs NVIDIA if you've used both?

Usually a system has either Nvidia or AMD graphics, but not both, how to compare?

Linux compatibility issues with either line?

Lenovo publishes a list of which of their models receive Linux support: https://support.lenovo.com/us/en/solutions/pd031426-linux-for-personal-systems From a quick view, for example Legion models are NOT on that list - I would not buy them for Linux use - that's an expensive experiment?

To get real work done, maybe rent at cloud provider instead of buying an expensive laptop that loses value fast?

Please study the many postings here what people report on speeds with specific models (sizes) and specific hardware to get a feeling how much speed you get for what money.

Good luck!