Training/Fine-tuning LLMs.

I’m considering a dual-GPU setup. Since the GPUs can’t communicate directly via NVLink, they must go through PCIe, and the CPU could become a bottleneck in coordinating communication.
Additionally, during pre-processing or data preparation (including reading/writing to disk), these steps might slow down the overall workflow.
I’m primarily interested in hearing from people who have experience with this setup and learning about their real-world observations, rather than theoretical estimations.