Two separate Hard Drives corrupted in as many days... User error?

archomrade [he/him] · edit-2 7 months ago

Two separate Hard Drives corrupted in as many days... User error?

@mortrek · 7 months ago

If you are getting actual hardware/sata errors on the host (not sure if that’s exactly what’s happening from your description), and multiple drives have had a similar problem, I’d suspect the sata cable or controller/mobo. Intel had a lot of weird sata issues on their older chipsets, so I’d also recommend making sure it has the latest bios update. Could you be more specific on what kind of hardware errors are showing up? Like, maybe parts of the logs.

Human Crayon · 7 months ago

Came here to say I had something similar happen with my NAS a year back. Thought it was the drives, then the controller it was attached to. Turns out it was some crappy blue breakout cables causing the drives to error out and disconnect.

Ordered new breakout cables of a different brand and have zero errors since.

archomrade [he/him] · 7 months ago

i’m going back and looking, but I may have deleted logs for the VM’s when I deleted the VM and started repair.

here’s a readout of one of the instances of trying to shutdown the VM and having to ssh in and ‘force’ a shutdown (didn’t think i was forcing it from the terminal window, but maybe I did?) Doens’t give much more information.

/var/log/pve/tasks/D/UPID:pve1:00000AD5:0000AEFC:652DD26D:qmshutdown:110:root@pam::TASK ERROR: VM quit/powerdown failed - got timeout

i’m still looking for more detailed logs, but i’m starting to wonder if you’re right. This makes me more sad than having messed something up myself, because fixing it would involve buying more hardware :(

oop, just found some better ones in the journalctl. These were happening way earlier than I thought:

Oct 13 17:28:42 pve1 kernel: SMB2_read: 36 callbacks suppressed
Oct 13 17:28:42 pve1 kernel: CIFS: VFS: Send error in read = -5
Oct 13 17:28:42 pve1 kernel: CIFS: Status code returned 0xc0000185 STATUS_IO_DEVICE_ERROR
Oct 13 17:28:42 pve1 kernel: CIFS: VFS: Send error in read = -5
Oct 13 17:28:42 pve1 kernel: CIFS: Status code returned 0xc0000185 STATUS_IO_DEVICE_ERROR

there’s a million of these.

and here are some of the one’s i was seeing when I popped open the console while it was happening. pve1 was the mount device to the VM running the OMV server I think:

Oct 14 23:14:14 pve1 kernel: I/O error, dev sdb, sector 1801348384 op 0x0:(READ) flags 0x1000000 phys_seg >

and a bunch of these, looks like they happen after a lot of I/O errors happen and the system can’t reach the smb server anymore:

Oct 16 13:25:04 pve1 kernel: CIFS: VFS: \192.168.0.135\plex_media BAD_NETWORK_NAME: \192.168.0.135\plex_media

Here’s ones from yesterday, probably around the time i was getting the new HDD back up again. These call out the sata port specifically, and it’s running repeatedly in a loop:

Oct 18 21:52:22 pve1 kernel: ata4.00: configured for UDMA/133
Oct 18 21:52:22 pve1 kernel: ata4: EH complete
Oct 18 21:52:22 pve1 kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 21:52:22 pve1 kernel: ata4.00: irq_stat 0x40000001
Oct 18 21:52:22 pve1 kernel: ata4.00: failed command: WRITE DMA EXT
Oct 18 21:52:22 pve1 kernel: ata4.00: cmd 35/00:a8:00:38:ff/00:00:27:01:00/e0 tag 18 dma 86016 out
                                      res 51/04:a8:00:38:ff/00:00:27:01:00/e0 Emask 0x1 (device error)
Oct 18 21:52:22 pve1 kernel: ata4.00: status: { DRDY ERR }
Oct 18 21:52:22 pve1 kernel: ata4.00: error: { ABRT }

and here’s some more implicating the sdc device:

Oct 18 21:57:23 pve1 kernel: sd 3:0:0:0: [sdc] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
Oct 18 21:57:23 pve1 kernel: sd 3:0:0:0: [sdc] tag#25 Sense Key : Illegal Request [current]
Oct 18 21:57:23 pve1 kernel: sd 3:0:0:0: [sdc] tag#25 Add. Sense: Unaligned write command
Oct 18 21:57:23 pve1 kernel: sd 3:0:0:0: [sdc] tag#25 CDB: Write(16) 8a 00 00 00 00 00 e8 c4 08 00 00 00 00 08 00 00
Oct 18 21:57:23 pve1 kernel: I/O error, dev sdc, sector 3905161216 op 0x1:(WRITE) flags 0x1008800 phys_seg 1 prio class 2

There are actually kind of painting a picture. The culprit looks like that sata port, i’ll see if i can switch it to another and do some test writes, maybe that’ll fix it

Atemu · 7 months ago

The CIFS errors and logs inside the VMs are rather uninteresting as they’re just passing through the underlying HW’s issue.

These logs presented here definitely indicate an issue between CPU and drives. Could also be RAM but I’d check SATA cables and controllers first.

archomrade [he/him] · edit-2 7 months ago

Yup, after scrubbing the log file, the problem port is ONLY ATA port 4.00. No other ports have thrown errors, BUT, i just did a block check on all the boot drive partitions, and it looks like they all have bad superblocks… not sure if the issue then is with the specific sata port or if the issue originates in the memory, or if the bad blocks get propagated to the other drives? unclear.

Oct 19 09:59:17 pve1 kernel: ata4.00: cmd 35/00:08:00:08:c4/00:00:e8:00:00/e0 tag 15 dma 4096 out
Oct 19 09:59:17 pve1 kernel: ata4.00: status: { DRDY ERR }
Oct 19 09:59:17 pve1 kernel: ata4.00: error: { ABRT }
Oct 19 09:59:17 pve1 kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 19 09:59:17 pve1 kernel: ata4.00: irq_stat 0x40000001
Oct 19 09:59:17 pve1 kernel: ata4.00: failed command: WRITE DMA EXT

i’ll do:

a memory test
swap the ports of the HD’s to specifically avoid port4.00
do a read-write test to make sure the issue doesn’t re-appear.

if non of the above solves the mystery, i suppose I can splurge on another junker and see if I have better luck on the next one. I just have to decide if I wait for ddrescue to finish, or just start it now… Probably start it now, on the off-chance i’m just creating more bad blocks on the backup.

Atemu · 7 months ago

do a read-write test to make sure the issue doesn’t re-appear

I can recommend https://wiki.archlinux.org/title/Badblocks

archomrade [he/him] · 7 months ago

fuck me, that test was damning. Read test was fine, but starting a read-write test revealed all the same I/O errors as before, this time on a differen’t port.

@mortrek · 7 months ago

Have you tried a running a different distro live f/usb or something like that? Doesn’t seem likely that it would help, but who knows…

It’s unlikely the kernel or other low-level code is the problem on 10 year old Intel hardware, though. I’ve run numerous distros on numerous different machines, many of which were Intel-based, over the last couple decades, and never had this kind of basic, low-level problem with SATA before without it being the cable or controller. Oh, I just remembered: check the PSU as well if you can. A faulty PSU could have a bad rail or wire or something that leads to these problems. If you have a known-good one lying around, depending on the motherboard, you could try temporarily hooking it up to the board and drive and see if it changes anything.

To eliminate Linux as a potential culprit, you could try to install Windows (7, 8, 10, whatever) and see if it exhibits similar problems.

archomrade [he/him] · 7 months ago

Well shit. Looks like the other sata ports are having the same problem.

trying to get a hardware probe running, but what are the chances i need to replace the motherboard/the machine? It’s looking likely the problem is upstream from the sata drives themselves, i just don’t know if it’s worth trying to swap the cpu before just ditching the machine entirely. I don’t have a cpu lying around to test it. memtester came back clean after 5 passes.

Atemu · 7 months ago

I’d honestly just abandon the hardware. It’s not worth your time to deal with that.