My disk fails to mount after a reboot. What do I do?
I am running into an issue with these 2 servers that is preventing me from mounting a remote nfs-mount, hosted on another linode. I have verified that that linode is operating just fine by mounting the nfs on my local machine. Every time I reboot the machine it goes into recovery mode and it fails to mount. Any help here?
First things first, if you are reaching out because of our recent fleet-wide Spectre maintenance, which involved host reboots, please also check out the two links below:
Further to the recommendations above, when you try to reboot, please pay attention to the boot sequence. If there are any errors, please take note of them. If you are unable to boot, try first to perform a filesystem check to pinpoint the issue. You’ll need to do this in Rescue Mode, which is only accessible in the Linode Manager.
To access the Rescue function of the relevant Linode, you would select the Rescue tab for the Linode. In the Rescue Mode section, select the disks you want to be mounted. (Make a note of where the disks are located. You will need this information later.) Once booted in Rescue Mode, you will find that none of your Linode's disks are mounted yet.
They're accessible, but they remain unmounted so you can safely run filesystem checks on them. For reference on using Rescue Mode:
Click Reboot into Rescue Mode. The Linode’s dashboard appears. Watch the Host Job Queue on the Dashboard to monitor the progress. To access your Linode when it’s running in rescue mode, you’ll need to use the Linode shell (Lish). To access Lish, click the link below:
Once you’ve successfully logged in, you’ll be able to run the following command for the FSCK:
e2fsck -f /dev/sda
(Be sure to substitute /dev/sda for the correct block device.)
Once you've run the filesystem check, if there are no errors, you’ll want to mount the disk to assess any of the errors encountered during the boot process. You can do this by running this command:
mount -o exec,barrier=0 /dev/sda
If you'd rather, you can chroot into it to run a shell from your Linode's data disk. After mounting:
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
mount -t devpts pts dev/pts/
chroot /media/sda /bin/bash
Most often, with mount issues, the errors you’d be looking for would be ‘Filesystem Dependency’ related. If you notice any volumes causing these types of errors during boot, you’ll want to check your /etc/fstab for correlation. Essentially, when a volume is removed but the entry is not removed from the fstab, it can cause mount issues. Once your disk is mounted, open /etc/fstab with your preferred text editor:
vi /etc/fstab/ or
Look for the volume in question, and comment out or remove the line that references the volume. Once you've done that, save the file, reboot out of Rescue Mode and see if the issue is resolved. Certainly, there could be a plethora of additional reasons for mount issues post-reboot, but this is one that we see customers run into most often. If anyone has any other scenarios or troubleshooting steps they’d like to add, please feel free to share.
Final step: Exit from Rescue mode and log back into your server.