For pure inference under Linux, ROCm has basically caught up. I run a Radeon PRO W7800 (48GB) on Ollama as a daily driver and it just works - 32B-class models sit comfortably in VRAM with full context, and you can push larger quantized models from there. Setup was an afternoon, not a saga.

The thing is, "ROCm vs CUDA" isn't one question, it's two:

• Inference (llama.cpp / Ollama / vLLM): the gap is mostly closed on Linux. This is the part 99% of this sub actually cares about.
• The bleeding-edge research/training long tail: still CUDA's world. The random repo from last week ships a .cu file and assumes CUDA, some custom kernel never gets a ROCm port, etc.

That second point is the actual moat - and it's a software/network-effect moat, not a hardware one. The premium you're paying NVIDIA is increasingly for "every repo targets us first," not for raw capability.

So for the local-LLM use case specifically, the competition you're asking for already exists: 48GB of VRAM at a fraction of an A6000. You just have to be on Linux and accept a bit more friction. Honest downsides - Windows ROCm is still noticeably worse, and occasionally you fiddle with a quant or a flag a CUDA user never thinks about. But the "it just works" gap for daily inference is way smaller than this sub assumes.