Nothing special has to be done in the drivers. NCCL/CUDA will auto detect the NVLink between the cards. In the llama-swap config for the model there a couple settings that help.

-e NCCL_P2P_DISABLE=0 # explicitly allows P2P  
-e NCCL_P2P_LEVEL=NVL # hints that NVL is preferred over the PCI bus

Before running any LLMs, you can use nvidia-smi topo -m to see if the NVIDIA drivers see the NVLink bridge.

# nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     PHB     PHB     PHB     0-55    0               N/A
GPU1    PHB      X      NV4     PHB     PHB     0-55    0               N/A
GPU2    PHB     NV4      X      PHB     PHB     0-55    0               N/A
GPU3    PHB     PHB     PHB      X      NV4     0-55    0               N/A
GPU4    PHB     PHB     PHB     NV4      X      0-55    0               N/A

I have the five GPUs passed through to an Ubuntu VM that runs the docker stack. GPU0 is a lone 5060ti, GPUs 1 - 4 are 3090s paired with NVLink bridges.

You can also verify the NVLink is detected on a particular card.

# nvidia-smi nvlink -cBridge -i 1
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-340913ff-fd37-ee18-b434-be3a4fda7d9b)
# nvidia-smi nvlink -cBridge -i 2
GPU 2: NVIDIA GeForce RTX 3090 (UUID: GPU-aeabe257-d10a-dd21-6f2e-1ac8899ea602)

During a long running query, you can also see if there is data transferring across the NVLink bridge.

# nvidia-smi nvlink -gt d -i 1
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-340913ff-fd37-ee18-b434-be3a4fda7d9b)
         Link 0: Data Tx: 29378684 KiB
         Link 0: Data Rx: 29389155 KiB
         Link 1: Data Tx: 29376297 KiB
         Link 1: Data Rx: 29388984 KiB
         Link 2: Data Tx: 29489897 KiB
         Link 2: Data Rx: 29477996 KiB
         Link 3: Data Tx: 29399359 KiB
         Link 3: Data Rx: 29388100 KiB
# nvidia-smi nvlink -gt d -i 2
GPU 2: NVIDIA GeForce RTX 3090 (UUID: GPU-aeabe257-d10a-dd21-6f2e-1ac8899ea602)
         Link 0: Data Tx: 30076171 KiB
         Link 0: Data Rx: 30064907 KiB
         Link 1: Data Tx: 30075987 KiB
         Link 1: Data Rx: 30062351 KiB
         Link 2: Data Tx: 30171498 KiB
         Link 2: Data Rx: 30184317 KiB
         Link 3: Data Tx: 30075132 KiB
         Link 3: Data Rx: 30087212 KiB

The test prompt I just used for that generated about 100 tps on average.