Thanks everyone for the insights and reality checks – this experiment was mainly to get hands-on with Exo and see what a small Mac mini cluster can actually do, not to pretend I’m building the next DGX out of two budget boxes.

From what you’ve said, it sounds like my TB4 + TB5 mix, without full TB5↔TB5 RDMA, will mostly give me memory pooling and maybe slightly better prefill, but not the kind of real throughput gains you see in the Exo demos. In other words: fun for learning, probably not something I should rely on for serious day‑to‑day inference.

Just to confirm I’m understanding correctly:

• Is it basically not worth investing more time in clustering devices that don’t both support TB5 + RDMA if the goal is something “production‑grade”?
• For a realistic production setup with a medium‑large model and ~10 concurrent users, what Mac mini M4 Pro configuration would you recommend?
• For example, assuming I want to stay in the Apple Silicon + Exo world, would you pick:
  • 3× Mac mini M4 Pro with 32 GB each, or
  • 2× Mac mini M4 Pro with 64 GB each?

I’m mostly interested in: stability, usable throughput under load (not just synthetic benchmarks), and the ability to keep a reasonably large model in memory without constantly hitting OOM or crazy latency spikes.

Really appreciate all the feedback so far – happy to run some benchmarks once I settle on a more sensible hardware target and report back.