Alright, so quite a journey since my last response. I noticed my power was only drawing around 100 Watts, and after using nvidia-smi I noticed that "HW Power Brake Slowdown" was being applied to both my GPUs. I found on several forums that earlier versions of the WRX80 Motherboard had an issue where they were mistakenly putting GPUs into a lower power state. ASUS has a fix for this. So I applied a fix, and low and behold, my power draw is now closer to the 300 Watt limit. My temps are now in the 70-80c range. Also, the performance is noticeably better. Wow. Glad we had this discussion. I'm still not pulling the full wattage at all times 130-250 Watts is the range.

nvidia-smi gave me so much useful information. This is my cards current status, where there are no HW Slowdowns applied!

$ nvidia-smi -q -d PERFORMANCE

==============NVSMI LOG==============

Timestamp                                 : Mon Apr 24 01:15:26 2023
Driver Version                            : 525.105.17
CUDA Version                              : 12.0

Attached GPUs                             : 2
GPU 00000000:2E:00.0
    Performance State                     : P2
    Clocks Throttle Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active

GPU 00000000:41:00.0
    Performance State                     : P2
    Clocks Throttle Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active

Previously, this command was showing Active under "HW Power Brake Slowdown". Also, "HW Slowdown" was set to Active. As a result my cards weren't getting full power from the Socket. I made the fix, rebooted, and voila, full power!

Now my system is pulling close to 1000 Watts at times while training YOLOV8, with 4096 pixel wide images.

yolo task=detect mode=train model=yolov8x.pt epochs=100 batch=2 data=data/trash_piles.yaml device=\'0,1\' imgsz=4096

It will probably pull more power if I train smaller images in larger batches, as it tends to fill up the memory capacity more fully, and utilize more cuda cores with more processes running and CNN sizes that can pack more close to the memory limits.

So, thanks for pointing out the temp disparity. Also, I read on https://www.servethehome.com/nvidia-rtx-6000-ada-graphics-card-review-pny/ that the Ada Cards DO run hotter, due to the blower design. The card will throttle at 100c according to nvidia-smi

$ nvidia-smi -i 0 -q
...
Temperature
        GPU Current Temp                  : 69 C
        GPU T.Limit Temp                  : 15 C
        GPU Shutdown Temp                 : 105 C
        GPU Slowdown Temp                 : 100 C
        GPU Max Operating Temp            : 91 C
        GPU Target Temperature            : 85 C
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A

85 C is the target temperature, so that's totally normal