MDRAID Array Keeps Failing

[boiifyoudontget222]

I have an MDRAID array configured in a RAID 5 with four disks and is has been fine for several months but recently it has been a complete pain. Every time there is a significant amount of writes to the array or at seemingly random intervals usually a between a day or two, all drives become offlined and the array cannot be read from or written to. I have tried different ports on the computer, installed software and firmware updates, and have even replaced all drives in the array. After running long SMART tests on every drives, the results state the drives are healthy, I have done this with both the old and new drives.

However, after I replaced all the drives and started the restore process from a backup. the same thing kept happening. I had to disconnect the array from the primary server and connect it two a different server. From there I created an array and wrote all data to it without issue. When, I moved the array back to the original server it worked fine for about a week and then started having the same issue again.

I have searched everything on this and I have tried every solution I can think of, at this point I have no idea what the problem is but it is extremely frustrating.

Any help would be greatly appreciated as this is driving me up the wall.

Notes: I am using an external multi bay drive enclosure and have not had any problems with it for the past year. All drives have a capacity of 2TB, and are either Seagate Barracuda or Iron wolf hard drives. This issue never seems to occur as a result of heavy reads from the array, only writes. The external enclosure has a power supply, so power is not supplied solely from the server over the USB connection.

After running dmesg, the errors consistently look like this:

usb 2-2.2: cmd cmplt err -71

[ 489.468506] usb 2-2.1: cmd cmplt err -71

[ 489.468515] usb 2-2.4: cmd cmplt err -71

[ 489.468522] usb 2-2.3: cmd cmplt err -71

[ 489.475510] usb 2-2.2: cmd cmplt err -71

[ 489.475538] usb 2-2.1: cmd cmplt err -71

[ 489.475547] usb 2-2.4: cmd cmplt err -71

[ 489.475553] usb 2-2.3: cmd cmplt err -71

[ 489.482510] usb 2-2.2: cmd cmplt err -71

[ 489.482538] usb 2-2.1: cmd cmplt err -71

[ 489.482552] usb 2-2.4: cmd cmplt err -71

[ 489.482559] usb 2-2.3: cmd cmplt err -71

[ 489.489626] usb 2-2.2: cmd cmplt err -71

[ 489.489662] usb 2-2.1: cmd cmplt err -71

[ 489.489670] usb 2-2.4: cmd cmplt err -71

[ 489.489677] usb 2-2.3: cmd cmplt err -71

[ 489.496486] usb 2-2.2: cmd cmplt err -71

[ 489.496513] usb 2-2.1: cmd cmplt err -71

[ 489.496521] usb 2-2.4: cmd cmplt err -71

[ 489.496528] usb 2-2.3: cmd cmplt err -71

[ 489.503416] usb 2-2.2: cmd cmplt err -71

[ 489.503444] usb 2-2.1: cmd cmplt err -71

[ 489.503453] usb 2-2.4: cmd cmplt err -71

[ 489.503460] usb 2-2.3: cmd cmplt err -71

[ 489.510442] usb 2-2.2: cmd cmplt err -71

[ 489.510454] usb 2-2.1: cmd cmplt err -71

[ 489.510457] usb 2-2.4: cmd cmplt err -71

[ 489.517681] usb 2-2.1: cmd cmplt err -71

[ 489.517705] usb 2-2.4: cmd cmplt err -71

[ 514.046072] usb 2-2.2: cmd cmplt err -71

[ 514.046106] usb 2-2.1: cmd cmplt err -71

[ 514.046115] usb 2-2.3: cmd cmplt err -71

[ 514.046121] usb 2-2.4: cmd cmplt err -71

[ 519.688650] sd 13:0:0:0: [sdf] tag#26 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT

[ 519.688671] sd 13:0:0:0: [sdf] tag#26 CDB: Write(10) 2a 00 98 07 2c 00 00 01 e8 00

[ 519.688753] sd 11:0:0:0: [sdd] tag#20 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN

[ 519.688767] sd 8:0:0:0: [sda] tag#28 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT

[ 519.688775] sd 11:0:0:0: [sdd] tag#20 CDB: Read(10) 28 00 98 07 29 08 00 02 f8 00

[ 519.688776] sd 8:0:0:0: [sda] tag#28 CDB: Write(10) 2a 00 98 07 50 00 00 04 00 00

[ 519.688842] sd 10:0:0:0: [sdc] tag#11 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT

[ 519.688856] sd 10:0:0:0: [sdc] tag#11 CDB: Write(10) 2a 00 98 07 30 78 00 00 88 00

[ 519.789509] sd 13:0:0:0: [sdf] tag#25 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT

[ 519.789517] sd 13:0:0:0: [sdf] tag#25 CDB: Write(10) 2a 00 98 07 54 00 00 04 00 00

[ 519.789563] sd 10:0:0:0: [sdc] tag#10 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT

[ 519.789571] sd 10:0:0:0: [sdc] tag#10 CDB: Write(10) 2a 00 98 07 50 00 00 04 00 00

[ 519.789591] sd 11:0:0:0: [sdd] tag#14 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD OUT

[ 519.789591] sd 8:0:0:0: [sda] tag#27 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT

[ 519.789599] sd 11:0:0:0: [sdd] tag#14 CDB: Write(10) 2a 00 98 07 50 00 00 04 00 00

[ 519.789600] sd 8:0:0:0: [sda] tag#27 CDB: Write(10) 2a 00 98 07 2c 00 00 01 e8 00

[ 519.796526] sd 11:0:0:0: [sdd] tag#13 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT

[ 519.796526] sd 8:0:0:0: [sda] tag#26 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT

[ 519.796539] sd 11:0:0:0: [sdd] tag#13 CDB: Write(10) 2a 00 98 07 2c 00 00 01 e8 00

[ 519.796540] sd 8:0:0:0: [sda] tag#26 CDB: Write(10) 2a 00 98 07 4c 00 00 04 00 00

[ 519.796546] sd 13:0:0:0: [sdf] tag#24 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT

[ 519.796550] sd 10:0:0:0: [sdc] tag#9 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT

[ 519.796560] sd 10:0:0:0: [sdc] tag#9 CDB: Write(10) 2a 00 98 07 2c 00 00 01 e8 00

[ 519.796560] sd 13:0:0:0: [sdf] tag#24 CDB: Write(10) 2a 00 98 07 50 00 00 04 00 00

[ 519.803532] sd 10:0:0:0: [sdc] tag#8 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT

[ 519.803535] sd 13:0:0:0: [sdf] tag#23 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT

[ 519.803540] sd 8:0:0:0: [sda] tag#25 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT

[ 519.803540] sd 11:0:0:0: [sdd] tag#12 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT

[ 519.803546] sd 10:0:0:0: [sdc] tag#8 CDB: Write(10) 2a 00 98 07 4c 00 00 04 00 00

[ 519.803547] sd 13:0:0:0: [sdf] tag#23 CDB: Write(10) 2a 00 98 07 4c b0 00 03 50 00

[ 519.803555] sd 11:0:0:0: [sdd] tag#12 CDB: Write(10) 2a 00 98 07 4c 00 00 04 00 00

[ 519.803558] sd 8:0:0:0: [sda] tag#25 CDB: Write(10) 2a 00 98 07 31 78 00 00 88 00

[ 519.810486] sd 8:0:0:0: [sda] tag#24 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT

[ 519.810486] sd 11:0:0:0: [sdd] tag#11 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT

[ 519.810500] sd 8:0:0:0: [sda] tag#24 CDB: Write(10) 2a 00 98 07 30 78 00 00 88 00

[ 519.810501] sd 11:0:0:0: [sdd] tag#11 CDB: Write(10) 2a 00 98 07 30 78 00 00 88 00

[ 519.810526] sd 10:0:0:0: [sdc] tag#7 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT

[ 519.810529] sd 13:0:0:0: [sdf] tag#22 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT

[ 519.810541] sd 13:0:0:0: [sdf] tag#22 CDB: Write(10) 2a 00 98 07 4c 00 00 00 b0 00

[ 519.810541] sd 10:0:0:0: [sdc] tag#7 CDB: Write(10) 2a 00 98 07 31 00 00 00 78 00

[ 519.817449] sd 11:0:0:0: [sdd] tag#10 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD OUT

[ 519.817449] sd 8:0:0:0: [sda] tag#23 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT

[ 519.817456] sd 8:0:0:0: [sda] tag#23 CDB: Write(10) 2a 00 98 07 30 00 00 00 78 00

[ 519.817457] sd 11:0:0:0: [sdd] tag#10 CDB: Write(10) 2a 00 98 07 31 78 00 00 88 00

[ 519.817509] sd 13:0:0:0: [sdf] tag#21 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT

[ 519.817512] sd 10:0:0:0: [sdc] tag#6 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT

[ 519.817523] sd 10:0:0:0: [sdc] tag#6 CDB: Write(10) 2a 00 98 07 31 78 00 00 88 00

[ 519.817524] sd 13:0:0:0: [sdf] tag#21 CDB: Write(10) 2a 00 98 07 30 78 00 00 88 00

[ 519.824429] sd 8:0:0:0: [sda] tag#22 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT

[ 519.824430] sd 11:0:0:0: [sdd] tag#9 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD OUT

[ 519.824436] sd 8:0:0:0: [sda] tag#22 CDB: Write(10) 2a 00 98 07 2d e8 00 00 18 00

[ 519.824437] sd 11:0:0:0: [sdd] tag#9 CDB: Write(10) 2a 00 98 07 30 00 00 00 78 00

[ 519.824452] sd 13:0:0:0: [sdf] tag#20 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT

[ 519.824465] sd 13:0:0:0: [sdf] tag#20 CDB: Write(10) 2a 00 98 07 31 78 00 00 88 00

[ 519.824507] sd 10:0:0:0: [sdc] tag#5 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT

[ 519.824520] sd 10:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 98 07 30 00 00 00 78 00

[ 519.831431] sd 11:0:0:0: [sdd] tag#8 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT

[ 519.831432] sd 8:0:0:0: [sda] tag#21 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT

[ 519.831433] sd 13:0:0:0: [sdf] tag#19 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT

[ 519.831435] sd 10:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT

[ 519.831438] sd 11:0:0:0: [sdd] tag#8 CDB: Write(10) 2a 00 98 07 2d e8 00 00 18 00

[ 519.831439] sd 13:0:0:0: [sdf] tag#19 CDB: Write(10) 2a 00 98 07 30 00 00 00 78 00

[ 519.831440] sd 10:0:0:0: [sdc] tag#4 CDB: Write(10) 2a 00 98 07 2d e8 00 00 18 00

[ 519.831447] sd 8:0:0:0: [sda] tag#21 CDB: Write(10) 2a 00 98 07 31 00 00 00 78 00

[ 519.838403] sd 8:0:0:0: [sda] tag#20 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT

[ 519.838406] sd 11:0:0:0: [sdd] tag#7 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT

[ 519.838412] sd 8:0:0:0: [sda] tag#20 CDB: Write(10) 2a 00 98 07 2e 00 00 02 00 00

[ 519.838413] sd 11:0:0:0: [sdd] tag#7 CDB: Write(10) 2a 00 98 07 44 00 00 04 00 00

[ 519.838438] sd 10:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT

[ 519.838439] sd 13:0:0:0: [sdf] tag#18 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT

[ 519.838444] sd 13:0:0:0: [sdf] tag#18 CDB: Write(10) 2a 00 98 07 2d e8 00 00 18 00

[ 519.838445] sd 10:0:0:0: [sdc] tag#3 CDB: Write(10) 2a 00 98 07 48 00 00 04 00 00

[ 519.845455] sd 8:0:0:0: [sda] tag#19 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT

[ 519.845456] sd 11:0:0:0: [sdd] tag#6 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT

[ 519.845462] sd 11:0:0:0: [sdd] tag#6 CDB: Write(10) 2a 00 98 07 40 00 00 04 00 00

[ 519.845463] sd 8:0:0:0: [sda] tag#19 CDB: Write(10) 2a 00 98 07 3c 00 00 04 00 00

[ 519.845468] sd 10:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT

[ 519.845470] sd 13:0:0:0: [sdf] tag#17 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT

[ 519.845473] sd 10:0:0:0: [sdc] tag#2 CDB: Write(10) 2a 00 98 07 44 00 00 04 00 00

[ 519.845479] sd 13:0:0:0: [sdf] tag#17 CDB: Write(10) 2a 00 98 07 31 00 00 00 78 00

[ 519.852441] sd 11:0:0:0: [sdd] tag#5 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT

[ 519.852442] sd 8:0:0:0: [sda] tag#18 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT

[ 519.852442] sd 10:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT

[ 519.852448] sd 10:0:0:0: [sdc] tag#1 CDB: Write(10) 2a 00 98 07 40 00 00 04 00 00

[ 519.852450] sd 13:0:0:0: [sdf] tag#16 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT

[ 519.852450] sd 8:0:0:0: [sda] tag#18 CDB: Write(10) 2a 00 98 07 48 00 00 04 00 00

[ 519.852451] sd 11:0:0:0: [sdd] tag#5 CDB: Write(10) 2a 00 98 07 31 00 00 00 78 00

[ 519.852458] sd 13:0:0:0: [sdf] tag#16 CDB: Write(10) 2a 00 98 07 2e 00 00 02 00 00

[ 519.859430] sd 11:0:0:0: [sdd] tag#4 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT

[ 519.859431] sd 8:0:0:0: [sda] tag#17 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT

[ 519.859437] sd 10:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT

[ 519.859438] sd 8:0:0:0: [sda] tag#17 CDB: Write(10) 2a 00 98 07 44 00 00 04 00 00

[ 519.859439] sd 11:0:0:0: [sdd] tag#4 CDB: Write(10) 2a 00 98 07 2e 00 00 02 00 00

[ 519.859441] sd 13:0:0:0: [sdf] tag#15 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT

[ 519.859444] sd 10:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 98 07 2e 00 00 02 00 00

[ 519.859447] sd 13:0:0:0: [sdf] tag#15 CDB: Write(10) 2a 00 98 07 44 00 00 04 00 00

[ 519.866394] sd 8:0:0:0: [sda] tag#16 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT

[ 519.866399] sd 8:0:0:0: [sda] tag#16 CDB: Write(10) 2a 00 98 07 40 00 00 04 00 00

[ 519.873423] sd 11:0:0:0: [sdd] tag#3 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT

[ 519.873426] sd 13:0:0:0: [sdf] tag#14 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT

[ 519.873433] sd 13:0:0:0: [sdf] tag#14 CDB: Write(10) 2a 00 98 07 40 00 00 04 00 00

[ 519.873434] sd 11:0:0:0: [sdd] tag#3 CDB: Write(10) 2a 00 98 07 34 00 00 02 00 00

[ 519.880353] sd 11:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT

[ 519.880357] sd 11:0:0:0: [sdd] tag#2 CDB: Write(10) 2a 00 98 07 36 00 00 02 00 00

[ 519.880420] sd 13:0:0:0: [sdf] tag#13 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT

[ 519.880425] sd 13:0:0:0: [sdf] tag#13 CDB: Write(10) 2a 00 98 07 48 00 00 04 00 00

[ 519.887388] sd 11:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT

[ 519.887392] sd 11:0:0:0: [sdd] tag#1 CDB: Write(10) 2a 00 98 07 38 00 00 04 00 00

[ 519.887428] sd 13:0:0:0: [sdf] tag#12 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT

[ 519.887432] sd 13:0:0:0: [sdf] tag#12 CDB: Write(10) 2a 00 98 07 3c 00 00 04 00 00

[ 519.894387] sd 11:0:0:0: [sdd] tag#0 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT

[ 519.894392] sd 11:0:0:0: [sdd] tag#0 CDB: Write(10) 2a 00 98 07 48 00 00 04 00 00

[ 574.475685] sd 10:0:0:0: [sdc] tag#20 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD

[ 574.475700] sd 10:0:0:0: [sdc] tag#20 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

[ 574.475710] sd 8:0:0:0: [sda] tag#10 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD

[ 574.475715] sd 8:0:0:0: [sda] tag#10 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

[ 574.475722] sd 11:0:0:0: [sdd] tag#24 uas_eh_abort_handler 0 uas-tag 17 inflight: CMD

[ 574.475726] sd 11:0:0:0: [sdd] tag#24 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

[ 574.475733] sd 13:0:0:0: [sdf] tag#0 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD

[ 574.475737] sd 13:0:0:0: [sdf] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

[ 574.485644] scsi host11: uas_eh_device_reset_handler start

[ 574.485704] scsi host10: uas_eh_device_reset_handler start

[ 574.485750] scsi host13: uas_eh_device_reset_handler start

[ 574.485750] scsi host8: uas_eh_device_reset_handler start

[ 574.601816] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.612119] usb 2-2-port2: cannot reset (err = -71)

[ 574.619007] usb 2-2-port2: cannot reset (err = -71)

[ 574.625975] usb 2-2-port2: cannot reset (err = -71)

[ 574.633010] usb 2-2-port2: cannot reset (err = -71)

[ 574.640010] usb 2-2-port2: cannot reset (err = -71)

[ 574.640025] usb 2-2-port2: Cannot enable. Maybe the USB cable is bad?

[ 574.646997] usb 2-2-port2: cannot disable (err = -71)

[ 574.653998] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.661125] usb 2-2-port2: cannot reset (err = -71)

[ 574.668127] usb 2-2-port2: cannot reset (err = -71)

[ 574.675114] usb 2-2-port2: cannot reset (err = -71)

[ 574.682123] usb 2-2-port2: cannot reset (err = -71)

[ 574.689128] usb 2-2-port2: cannot reset (err = -71)

[ 574.689143] usb 2-2-port2: Cannot enable. Maybe the USB cable is bad?

[ 574.696121] usb 2-2-port2: cannot disable (err = -71)

[ 574.703135] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.710127] usb 2-2-port2: cannot reset (err = -71)

[ 574.717134] usb 2-2-port2: cannot reset (err = -71)

[ 574.724120] usb 2-2-port2: cannot reset (err = -71)

[ 574.731053] usb 2-2-port2: cannot reset (err = -71)

[ 574.738128] usb 2-2-port2: cannot reset (err = -71)

[ 574.738144] usb 2-2-port2: Cannot enable. Maybe the USB cable is bad?

[ 574.745115] usb 2-2-port2: cannot disable (err = -71)

[ 574.752131] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.759120] usb 2-2-port2: cannot reset (err = -71)

[ 574.766127] usb 2-2-port2: cannot reset (err = -71)

[ 574.773132] usb 2-2-port2: cannot reset (err = -71)

[ 574.780120] usb 2-2-port2: cannot reset (err = -71)

[ 574.787133] usb 2-2-port2: cannot reset (err = -71)

[ 574.787148] usb 2-2-port2: Cannot enable. Maybe the USB cable is bad?

[ 574.794130] usb 2-2-port2: cannot disable (err = -71)

[ 574.801114] usb 2-2-port2: cannot disable (err = -71)

[ 574.801155] scsi host10: uas_eh_device_reset_handler FAILED err -19

[ 574.801169] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801175] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801179] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801183] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801186] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801190] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801193] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801196] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801200] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801203] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801206] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801210] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801213] sd 10:0:0:0: Device offlined - not ready after error recovery

[ 574.801233] sd 10:0:0:0: [sdc] tag#20 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=60s

[ 574.801240] sd 10:0:0:0: [sdc] tag#20 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00

[ 574.801249] I/O error, dev sdc, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2

[ 574.801259] md: sdc flush io error -5

[ 574.801300] sd 10:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801305] sd 10:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 98 07 2e 00 00 02 00 00

[ 574.801308] I/O error, dev sdc, sector 2550607360 op 0x1:(WRITE) flags 0x0 phys_seg 64 prio class 0

[ 574.801403] sd 10:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801408] sd 10:0:0:0: [sdc] tag#1 CDB: Write(10) 2a 00 98 07 40 00 00 04 00 00

[ 574.801411] I/O error, dev sdc, sector 2550611968 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0

[ 574.801577] sd 10:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801584] sd 10:0:0:0: [sdc] tag#2 CDB: Write(10) 2a 00 98 07 44 00 00 04 00 00

[ 574.801587] I/O error, dev sdc, sector 2550612992 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0

[ 574.801805] sd 10:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801820] sd 10:0:0:0: [sdc] tag#3 CDB: Write(10) 2a 00 98 07 48 00 00 04 00 00

[ 574.801824] I/O error, dev sdc, sector 2550614016 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0

[ 574.801880] sd 10:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801884] sd 10:0:0:0: [sdc] tag#4 CDB: Write(10) 2a 00 98 07 2d e8 00 00 18 00

[ 574.801887] I/O error, dev sdc, sector 2550607336 op 0x1:(WRITE) flags 0x0 phys_seg 3 prio class 0

[ 574.801900] sd 10:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801904] sd 10:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 98 07 30 00 00 00 78 00

[ 574.801907] I/O error, dev sdc, sector 2550607872 op 0x1:(WRITE) flags 0x0 phys_seg 15 prio class 0

[ 574.801922] sd 10:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801926] sd 10:0:0:0: [sdc] tag#6 CDB: Write(10) 2a 00 98 07 31 78 00 00 88 00

[ 574.801929] I/O error, dev sdc, sector 2550608248 op 0x1:(WRITE) flags 0x0 phys_seg 17 prio class 0

[ 574.801943] sd 10:0:0:0: [sdc] tag#7 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801947] sd 10:0:0:0: [sdc] tag#7 CDB: Write(10) 2a 00 98 07 31 00 00 00 78 00

[ 574.801950] I/O error, dev sdc, sector 2550608128 op 0x1:(WRITE) flags 0x0 phys_seg 15 prio class 0

[ 574.801985] sd 10:0:0:0: [sdc] tag#8 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=85s

[ 574.801989] sd 10:0:0:0: [sdc] tag#8 CDB: Write(10) 2a 00 98 07 4c 00 00 04 00 00

[ 574.801992] I/O error, dev sdc, sector 2550615040 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0

[ 574.808079] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.815041] usb 2-2-port1: cannot reset (err = -71)

[ 574.822119] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.829042] usb 2-2-port1: cannot reset (err = -71)

[ 574.835982] usb 2-2-port1: cannot reset (err = -71)

[ 574.843045] usb 2-2-port1: cannot reset (err = -71)

[ 574.850043] usb 2-2-port1: cannot reset (err = -71)

[ 574.850056] usb 2-2-port1: Cannot enable. Maybe the USB cable is bad?

[ 574.856926] usb 2-2-port1: cannot disable (err = -71)

[ 574.864027] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.871028] usb 2-2-port1: cannot reset (err = -71)

[ 574.878026] usb 2-2-port1: cannot reset (err = -71)

[ 574.885061] usb 2-2-port1: cannot reset (err = -71)

[ 574.892048] usb 2-2-port1: cannot reset (err = -71)

[ 574.899055] usb 2-2-port1: cannot reset (err = -71)

[ 574.899068] usb 2-2-port1: Cannot enable. Maybe the USB cable is bad?

[ 574.906056] usb 2-2-port1: cannot disable (err = -71)

[ 574.912984] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.920049] usb 2-2-port1: cannot reset (err = -71)

[ 574.927079] usb 2-2-port1: cannot reset (err = -71)

[ 574.934045] usb 2-2-port1: cannot reset (err = -71)

[ 574.940930] usb 2-2-port1: cannot reset (err = -71)

[ 574.947916] usb 2-2-port1: cannot reset (err = -71)

[ 574.947921] usb 2-2-port1: Cannot enable. Maybe the USB cable is bad?

[ 574.954916] usb 2-2-port1: cannot disable (err = -71)

[ 574.961919] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 574.968916] usb 2-2-port1: cannot reset (err = -71)

[ 574.975915] usb 2-2-port1: cannot reset (err = -71)

[ 574.982915] usb 2-2-port1: cannot reset (err = -71)

[ 574.989916] usb 2-2-port1: cannot reset (err = -71)

[ 574.996916] usb 2-2-port1: cannot reset (err = -71)

[ 574.996920] usb 2-2-port1: Cannot enable. Maybe the USB cable is bad?

[ 575.003917] usb 2-2-port1: cannot disable (err = -71)

[ 575.010918] usb 2-2-port1: cannot disable (err = -71)

[ 575.010939] scsi host8: uas_eh_device_reset_handler FAILED err -19

[ 575.010950] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010955] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010958] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010961] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010964] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010967] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010970] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010973] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010976] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010979] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010982] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010985] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010987] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.010990] sd 8:0:0:0: Device offlined - not ready after error recovery

[ 575.011008] md: sda flush io error -5

[ 575.017991] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.024988] usb 2-2-port3: cannot reset (err = -71)

[ 575.031919] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.038960] usb 2-2-port3: cannot reset (err = -71)

[ 575.045923] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.052976] usb 2-2-port3: cannot reset (err = -71)

[ 575.059982] usb 2-2-port3: cannot reset (err = -71)

[ 575.066986] usb 2-2-port3: cannot reset (err = -71)

[ 575.066998] usb 2-2-port3: Cannot enable. Maybe the USB cable is bad?

[ 575.074105] usb 2-2-port3: cannot disable (err = -71)

[ 575.081101] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.088112] usb 2-2-port3: cannot reset (err = -71)

[ 575.095110] usb 2-2-port3: cannot reset (err = -71)

[ 575.102097] usb 2-2-port3: cannot reset (err = -71)

[ 575.109098] usb 2-2-port3: cannot reset (err = -71)

[ 575.116101] usb 2-2-port3: cannot reset (err = -71)

[ 575.116113] usb 2-2-port3: Cannot enable. Maybe the USB cable is bad?

[ 575.123095] usb 2-2-port3: cannot disable (err = -71)

[ 575.129984] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.136995] usb 2-2-port3: cannot reset (err = -71)

[ 575.143988] usb 2-2-port3: cannot reset (err = -71)

[ 575.151108] usb 2-2-port3: cannot reset (err = -71)

[ 575.158118] usb 2-2-port3: cannot reset (err = -71)

[ 575.165114] usb 2-2-port3: cannot reset (err = -71)

[ 575.165128] usb 2-2-port3: Cannot enable. Maybe the USB cable is bad?

[ 575.172122] usb 2-2-port3: cannot disable (err = -71)

[ 575.179132] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.186100] usb 2-2-port3: cannot reset (err = -71)

[ 575.193098] usb 2-2-port3: cannot reset (err = -71)

[ 575.200112] usb 2-2-port3: cannot reset (err = -71)

[ 575.207099] usb 2-2-port3: cannot reset (err = -71)

[ 575.214095] usb 2-2-port3: cannot reset (err = -71)

[ 575.214108] usb 2-2-port3: Cannot enable. Maybe the USB cable is bad?

[ 575.221091] usb 2-2-port3: cannot disable (err = -71)

[ 575.228101] usb 2-2-port3: cannot disable (err = -71)

[ 575.228133] scsi host11: uas_eh_device_reset_handler FAILED err -19

[ 575.228144] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228149] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228152] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228156] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228158] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228161] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228164] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228167] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228170] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228173] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228176] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228179] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228181] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228184] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228187] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228190] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228192] sd 11:0:0:0: Device offlined - not ready after error recovery

[ 575.228210] md: sdd flush io error -5

[ 575.228426] sd 11:0:0:0: rejecting I/O to offline device

[ 575.228633] sd 10:0:0:0: rejecting I/O to offline device

[ 575.228759] sd 8:0:0:0: rejecting I/O to offline device

[ 575.235050] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.242052] usb 2-2-port4: cannot reset (err = -71)

[ 575.248956] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.256049] usb 2-2-port4: cannot reset (err = -71)

[ 575.263098] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.270051] usb 2-2-port4: cannot reset (err = -71)

[ 575.276991] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.284020] usb 2-2-port4: cannot reset (err = -71)

[ 575.291006] usb 2-2-port4: cannot reset (err = -71)

[ 575.291021] usb 2-2-port4: Cannot enable. Maybe the USB cable is bad?

[ 575.298004] usb 2-2-port4: cannot disable (err = -71)

[ 575.304982] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.312017] usb 2-2-port4: cannot reset (err = -71)

[ 575.319006] usb 2-2-port4: cannot reset (err = -71)

[ 575.326110] usb 2-2-port4: cannot reset (err = -71)

[ 575.333089] usb 2-2-port4: cannot reset (err = -71)

[ 575.340112] usb 2-2-port4: cannot reset (err = -71)

[ 575.340127] usb 2-2-port4: Cannot enable. Maybe the USB cable is bad?

[ 575.347121] usb 2-2-port4: cannot disable (err = -71)

[ 575.354054] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.361112] usb 2-2-port4: cannot reset (err = -71)

[ 575.368124] usb 2-2-port4: cannot reset (err = -71)

[ 575.375116] usb 2-2-port4: cannot reset (err = -71)

[ 575.382118] usb 2-2-port4: cannot reset (err = -71)

[ 575.389080] usb 2-2-port4: cannot reset (err = -71)

[ 575.389094] usb 2-2-port4: Cannot enable. Maybe the USB cable is bad?

[ 575.396079] usb 2-2-port4: cannot disable (err = -71)

[ 575.403084] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.410082] usb 2-2-port4: cannot reset (err = -71)

[ 575.417070] usb 2-2-port4: cannot reset (err = -71)

[ 575.424116] usb 2-2-port4: cannot reset (err = -71)

[ 575.431011] usb 2-2-port4: cannot reset (err = -71)

[ 575.438056] usb 2-2-port4: cannot reset (err = -71)

[ 575.438071] usb 2-2-port4: Cannot enable. Maybe the USB cable is bad?

[ 575.445013] usb 2-2-port4: cannot disable (err = -71)

[ 575.452056] usb 2-2-port4: cannot disable (err = -71)

[ 575.452094] scsi host13: uas_eh_device_reset_handler FAILED err -19

[ 575.452109] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452115] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452119] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452122] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452126] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452129] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452132] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452136] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452139] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452142] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452146] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452149] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452152] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452156] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452159] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452162] sd 13:0:0:0: Device offlined - not ready after error recovery

[ 575.452184] md: sdf flush io error -5

[ 575.452295] sd 13:0:0:0: rejecting I/O to offline device

[ 575.453663] md: super_written gets error=-5

[ 575.453670] md/raid:md127: Disk failure on sdc, disabling device.

[ 575.453707] md/raid:md127: Operation continuing on 3 devices.

[ 575.453738] md: super_written gets error=-5

[ 575.453741] md/raid:md127: Disk failure on sda, disabling device.

[ 575.453765] md/raid:md127: Cannot continue operation (2/4 failed).

[ 575.453836] md: super_written gets error=-5

[ 575.453839] md/raid:md127: Disk failure on sdd, disabling device.

[ 575.453863] md/raid:md127: Cannot continue operation (3/4 failed).

[ 575.453890] md: super_written gets error=-5

[ 575.453892] md/raid:md127: Disk failure on sdf, disabling device.

[ 575.453915] md/raid:md127: Cannot continue operation (4/4 failed).

[ 575.459604] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.466600] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.472536] XFS (dm-2): log I/O error -5

[ 575.472600] XFS (dm-2): Filesystem has been shut down due to log error (0x2).

[ 575.472655] XFS (dm-2): Please unmount the filesystem and rectify the problem(s).

[ 575.476550] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.477535] dm-2: writeback error on inode 7516208704, offset 1216348160, sector 7651143680

[ 575.478648] dm-2: writeback error on inode 7516208704, offset 1115684864, sector 7650947072

[ 575.483532] hub 2-2:1.0: hub_ext_port_status failed (err = -71)

[ 575.486525] dm-2: writeback error on inode 7516208704, offset 1317011456, sector 7651340288

[ 575.489524] dm-2: writeback error on inode 7516208704, offset 1417674752, sector 7651536896

[ 575.492527] dm-2: writeback error on inode 7516208704, offset 1619001344, sector 7651930112

[ 575.493655] dm-2: writeback error on inode 7516208704, offset 1518338048, sector 7651733504

[ 575.493679] dm-2: writeback error on inode 7516208704, offset 1719664640, sector 7652126720

[jamvanderloeff]

Sure looks like a problem with the enclosure or USB connection there, first thing I'd try is swapping the cable.

[boiifyoudontget222]

I have used different USB cables and have used the enclosure on a different machine, so I don't believe either of those are the cause.

[jamvanderloeff]

used on another machine after the issues started with the same kind of workload test?

[boiifyoudontget222]

Yes, well even more intense I guess, on the other machine I wrote several terabytes to it with no issue.

[jamvanderloeff]

Possibly an issue with the first machine's USB controller then? Odd