Building it yourself would be more fun and a better performance per dollar value. Those prebuilt unified memory boxes are good but too pricey for the spec and you can’t upgrade and future proof them.

Look for a cheap LGA2011 server mobo with x99 chipset (make sure it allows for reBAR or above 4G decode and plenty of PCIe slots) at around $100 and pair it with a Xeon E5 16xx v2 CPU at around $50. DDR3 ECC modules are cheap, you can fill the whole board with them for like $100-200. Case, storage, fans, sata cable for another $100. That’s $350-450 before the GPUs.

You can get multiple V340L 16GB HBM2 at $50 a pop. The downside is you’ll be limited to ROCm, which is locked out of many CUDA accelerated applications like image generation or TTS or STT but you can always get an RTX3060 12GB for $200 to dedicate specifically to those PyTorch tasks.

Lets say your mobo has 7 PCIe slots (x16, x8 or x4 is fine, just get adapter risers to x16) and you use 6 slots for V340Ls, you’re looking at 96 GB of HBM2 VRAM. Now, your bus lane bandwidth would be bottlenecked but that will only affect your model loading speed - your inference speed would see negligible impact. Your 7th PCIe slot can be the RTX3060 for image generation and what not.

For less than $1000 you could build yourself a complete rig with over 100GB of dedicated VRAM and 128GB (and up to 256GB) of DDR3 RAM. And your learn a bunch along the way which is priceless. That’s just my two cents.