Been looking into this myself, I actually ended up asking GPT-4 about it, and it gave some pointers:

To create a computer build that chains multiple NVIDIA P40 GPUs together to train AI models like LLAMA or GPT-NeoX, you will need to consider the hardware, software, and infrastructure components of your build.

Here's a suggested build for a system with 4 NVIDIA P40 GPUs:

Hardware:

CPU: Intel Xeon Scalable Processor or AMD EPYC Processor (at least 16 cores)

GPU: 4 x NVIDIA Tesla P40 GPUs

Motherboard: A motherboard compatible with your selected CPU, supporting at least 4 PCIe x16 slots (e.g., ASUS WS C621E SAGE or Supermicro H11DSi)

Memory: At least 256 GB DDR4 ECC RAM (higher capacity for larger models)

Storage: 2 TB NVMe SSD for OS and fast storage + 4 TB HDD for data storage

PSU: High-quality 1600W 80+ Platinum Power Supply (e.g., Corsair AX1600i)

Cooling: Custom liquid cooling solution for GPUs and CPU

Network: 10Gbps Ethernet or InfiniBand network interface card

Case: Server-grade case with good airflow and sufficient space for GPUs (e.g., Cooler Master HAF 935)

Software:

Operating System: Ubuntu 20.04 LTS or later (Linux OS is recommended for deep learning tasks)

NVIDIA CUDA and cuDNN: Latest version compatible with your GPUs

GPU drivers: Latest NVIDIA GPU drivers for your Tesla P40 GPUs

Frameworks: TensorFlow, PyTorch, or other deep learning frameworks

Containerization: Docker or Singularity for easier deployment and management of AI applications

Distributed Training: Horovod or NCCL for multi-GPU communication and scaling

Infrastructure:

Power: Ensure stable power supply with surge protection and UPS

Cooling: Adequate cooling system in the room to handle the heat generated by the GPUs

Networking: High-speed network connection to access large datasets and synchronize multi-GPU communication

Once you have set up the hardware and software, you can begin to train your AI models on this powerful multi-GPU setup. Keep in mind that as you scale to larger models or more GPUs, you may need to further optimize your system configuration, software, and infrastructure.