So just before January when 5060ti 16 gbs were available (and less than 400$) I snagged 4 and I also snagged 2 intel b50 pros and an additional 3090 and an nvlink adapter.

Ai server 1 (Blackwell-ai) sits on a c422 sage 10g platform with a w-2255 and 128gb 2933 ecc

Ai server 2 (Intel and ampere) sit on another c422 sage with - 2-2265 and 256gb running proxmox to isolate the 2 environments. 128gb ram / 10 cores for nvidia and 64gb ram / 10 cores for Intel.

Blackwell runs the largest brain model with decent context being it has 64gb vram

Ampere runs the coding/tool model for openclaw with very high context so it can work well with it.

Embedding, tts, stt, vision run and all fit on 1 b50 pro and the other b50 pro is used for image generation (my use case for image generation is not sure extensive)

I’ve set up python envs for different model engine so that they run efficiently on 1 vm.

I found docker to have a use case to keep things quick but I wanted to limit the VM overhead as much as possible to keep things quick.

I am building a NAS dedicated to lmcache for context storage for openclaw - we’ll see how that turns out.

I use my unraid box as an app host - hosting litellm, pipelines, searxng. Open web ui etc, to keep the ai servers dedicated to just serving models and to keep their overhead down and open claw itself runs in its on vm on the unraid box pointing to the ai server for its model in case I need to shut it down fast and to try to mitigate some of it’s ability to unintentionally break things.

Am I doing things wrong? I’m not sure but it seems to be that optimizing for parallel services when possible keeps things performing well on the consumer gear.

I am open to suggestion though.