Connection to external drives sometimes breaks on reboot
I've got a reoccurring issue with all of the home servers I've ever had and because it happened again just today, now the pain is big enough to ask publicly about it.
As of now, I'm running some Intel NUC ripoff with a JBOD attatched via USB 3, spinning a ZFS sort of-RAID. It's nothing that special tbh. In the past I had several other configurations with external drives, wired via fstab to Raspberry Pis and the like. All of those shared a similar issue: I can't recall exactly when, but I figure most of the time after updates to the kernel or docker, the computer(s) become stuck at boot. I had to unplug the external drives just to get the respective machine up, after which varying issues occurred with drives not being recognized anymore and such.
With my current setup, I run several docker containers which have their volumes on subdirectories/datasets on the /tank mountpoint, and when booting the machine without the drives, some of the containers create new directories at that destination, which now lives on my main drive /dev/sda.
It's not only painful to go through the manual process with the drives, I only have access the machine when I'm home, which I'm not all the time. Also, it's kind of time consuming as I'm backup up data that I fear might become inconsistent along the way. Every time I see a big kernel update, I fear that the computer will get stuck in such a situation once again and I'm reluctant to do a proper reboot.
I know that external drives are not best practice when it comes to handling "critical" data, but I don't want to run another machine just in order to provide access to the disks via network. Any ideas where these issues stem from and how to avoid them in the future?
Do you mount the drives using their /dev/sdX entries or via UUID? Because it sounds like you're using /dev/sdX entries (which you really shouldn't, because their names can randomly change, by design). Use /dev/disk/by-id/... directly for mounting or, alternatively, fill /etc/zfs/vdev_id.conf (see example below) and define the pool using their aliases.
alias Bay1 /dev/disk/by-id/ata-XXXXXXXXXXX1-YYYYY1_ZZZZZZZ4
alias Bay2 /dev/disk/by-id/ata-XXXXXXXXXXX2-YYYYY2_ZZZZZZZ4
alias Bay3 /dev/disk/by-id/ata-XXXXXXXXXXX3-YYYYY3_ZZZZZZZ4
alias Bay4 /dev/disk/by-id/ata-XXXXXXXXXXX4-YYYYY3_ZZZZZZZ4