@optimalwp is your Linode booting with Grub 2 by chance? We've been seeing reports of some Linodes have issues booting normally with Grub 2 – it seems to be related to an issue with the kernel scanning for SCSI devices asynchronously:

[    2.022709] sd 1:0:1:2: [sda] 1056768 512-byte logical blocks: (541 MB/516 MiB)
[    2.023789] sd 1:0:1:2: [sda] Write Protect is off
[    2.025452] sd 1:0:1:2: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.027243] sd 0:0:0:0: [sdb] 103817216 512-byte logical blocks: (53.2 GB/49.5 GiB)
[    2.028190] sd 0:0:0:0: [sdb] Write Protect is off
[    2.031940] sd 0:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.084375] sd 1:0:1:2: [sda] Attached SCSI disk
[    2.100267] sd 0:0:0:0: [sdb] Attached SCSI disk
mount: mounting /dev/sda on /root failed: No such device
Failed to mount /dev/sda as root file system.
(initramfs) [ 1723.420936] sd 1:0:1:2: [sda] Synchronizing SCSI cache
[ 1723.421704] sd 0:0:0

The easiest solution is going to be just switching to the Linode kernel:

https://www.linode.com/docs/guides/managing-the-kernel-on-a-linode/#updating-the-linode-kernel

Try selecting "Latest 64-bit" and rebooting, that should resolve the issue.

If you want to keep using Grub and the distro kernel, you'll need to modify your grub config (/etc/default/grub) by appending scsi_mod.scan=sync to the GRUB_CMDLINE_LINUX line, like so:

GRUB_CMDLINE_LINUX="console=ttyS0,19200n8 net.ifnames=0 scsi_mod.scan=sync"

Sorry like I said before, it's on the first boot after it's created, so maybe its the default setting?

@optimalwp yes understood – most of our distros deploy using Grub 2 by default, not the Linode kernel.

I just resized a Linode and bumped into this. Changing to Latest "64-bit" worked, but it's not great doing a resize and having to Google how to make it actually work. This should work OOB.

Yes, like @phplist I just hit the same issue after a resize and agree this should be handled better.

Changing the kernel worked for me.

I wanted follow up for other folks running into this.

After additional testing by several lovely users it's been discovered that the second solution of updating your grubconfig will not solve this problem in Debian 11 instances and you will continue to see intermittent boot failures.

At this time to resolve this issue for a Debian 11 instance you will want to switch to the Linode Kernel.

At this time to resolve this issue for a Debian 11 instance you will want to switch to the Linode Kernel.

I can confirm this solves the issue. Brand new Linode running Debian 11. error was as described above: Failed to mount /dev/sda as root file system.

Any chance Linode can fix the underlying issue?

If you use the Debian 11 disk image as provided by Linode, by default in /etc/default/grub the following line is uncommented:

GRUB_DISABLE_LINUX_UUID=true

Comment the line, by adding a # at the beginning:

#GRUB_DISABLE_LINUX_UUID=true

Save the file and run "update-grub".

With this change, during boot the kernel will not look for /dev/sda (or whichever disk you are using), but rather for the disk's UUID. So, even if the disk is on /dev/sdc, the kernel will find the correct disk.

A few more comments if you run into disks not assigned by Debian as configured in Linode.

/etc/fstab

Make sure to use LABEL= and UUID= in your /etc/fstab file instead of /dev/sd*. You can use the blkid command to find those. Example:

LABEL=root  /          ext4  noatime,nodiratime,errors=remount-ro 0 1
LABEL=bak   /bak       btrfs noatime,nodiratime                   0 2
UUID=e1b3ede9-e5b0-479d-835b-38419d290ff6 none swap noauto 0 0

The swap partition does not have a label, so there you must use UUID.

DRBD

If you happen to use DRBD: DRBD will not start if the disk is not found at the correct location (/dev/sdx). I created a udev rule and put it into a new file /etc/udev/rules.d/drbd.rules (it must have the .rules extension). I used ID_FS_UUID from the udevadm info /dev/sdx (replace sdx with the correct device name). You can find other attributes to uniquely identify the disk with udevadm info -a /dev/sdx.

Example:

SUBSYSTEM=="block", ENV{ID_FS_UUID}=="c9ea9dd696c0151", SYMLINK+="drbd0-disk"

After that you can use /dev/drbd0-disk.

Thank you, chroot! Your answer about commenting out that GRUB entry worked!