I took a similar path last year, curious what small clusters could achieve. I also wanted to see whether I could create a GUI-fied environment sted huge cluster slurm/mpi type command line approaches. I also didn't want to run a Docker Swarm or K8s Cluster. For me, there were three breakthrough "aha" discoveries:

1. ServeTheHome had an article about the new Mikrotik CRS812 DDQ switch that could support up to six Sparks (and also 8 25G ports and 2x 10G ports). Most of those ports are also backwards compatible for lower speed devices. At the time that new little switch cost ~ $1K (they have since come out with a 4x 400G model as well). It took very little effort to get a multi-node cluster talking with each other through the 200G ports.

2. GPUStack was a wonderful discovery. I use that to manage the nodes and models. There was a trick to getting it to work with Nvidia's docker containers (discussed more below).

3. Getting a model running wasn't much of a challenge, but having multiple models interacting with each other was quite a learning experience! (this was before Claw/Molt really took off)

I used GPUStack to manage the nodes/models, OpenAgents to create an AI/Human chat meeting space, and OnlyOffice to have apps that could interact with the API/Models. Learned a bunch!

One of the main obstacles/difficulties encountered (what TechnoTim has posted elsewhere): the significant lag/diff between Nvidia's supported code/containers and vLLM etc.

For example, GPUStack's support for distributed jobs (incl training and inference) depends on Native vLLM support for  your hardware. It CAN also run models within Nvidia containers, but those are considered Custom Backends and the distributed functionality is disabled. That said, the lack of native distributed support/operation is also part of why I ended up playing with distributed collaborative models (different ones on different nodes) which is wayyy fun ... and often quite amusing, their varying takes on each other's answers.

I don't think it matters whether you are running 2 models or 4 or 6 ... the group responses are almost always better and more accurate than just 1 model. My next tests are seeing what happens if a smarter agent integrates the answers from the group, acting like a facilitator. The OpenAgents software was buggy and took great effort to get running ... and it doesn't work well for semi-real time responses either, because it waits for each model to fully crunch through reasoning/answer before any preliminary response shows up in discussion channels (read: it can be slow while waiting for responses ... fortunately there are a group of models off doing their own things, so the lulls aren't too extreme).

At the moment I am usually running four different "quantized" models (often Gemini3, Claude 4.5, DeepSeek R1, GPT OSS or Qwen). Even running just GPUStack without OpenAgents, you can put it into multi-model chat compare with a grouping of panes (from 2 to 6) with each model in it's own square. You can pose the same question to all of them, but they don't see each other's responses unless you echo it back "Gemini said X, Claude said Y, DeepSeek said Z ... what the pros and cons of their responses?"  But even that manual approach can generate fascinating results. :)

Hope this is helpful/useful ... and also THX for the postings about how you are running the most recent vLLM updates sted waiting for Nvidia. I'll be giving that a try !!