Docker won't start using the latest Linode kernel

Avatar Linode 4 months ago

I recently rebooted my Linode and Docker will no longer start. I have my Linode configured to boot with the latest 64bit kernel. Prior to rebooting, Docker was starting without error. How do I fix this?

12 Replies

Updated 8.13.18:

Our team is aware of the issue with Docker and our recent kernels:
4.17.14-x86-linode132 and 4.17.14-x86_64-linode113

A specific module for OverlayFS was not present in these kernels which caused incompatibilities with Docker. This module is now included in our latest revisions, 4.17.14-x86-linode133 and 4.17.14-x86_64-linode114.

There are 2 easy solutions to get Docker back up and running:

  1. Reboot your Linode to boot with the latest Linode kernel.

Currently the latest 32bit and 64bit Linode kernels include support for OverlayFS.
Make sure your Linode is configured to boot using our Latest kernel within your configuration profile. This selection will allow your Linode to boot using the latest stable Linode kernel at each reboot.


  1. Use your distribution's upstream kernel.

Since all Linodes run on KVM, they are able to boot using either a Linode supplied kernel or your distribution's upstream kernel. You may have noticed that the newest distributions are already configured to use GRUB2 and your distribution's upstream kernel. This is the now the default method for all new Images and the preferred selection for kernels. To boot using a distribution supplied kernel, follow the steps in our guide here:
Running a Distribution-Supplied Kernel

Why would you deliberately release (and make default) a kernel which breaks a widely used piece of software, without announcing it?

It also breaks networking on a Debian 7 server I manage.

@turnip our team released a new set of 4.17.14 kernels today (linode114 and linode133) which should resolve this for you. If you are still seeing any Docker or networking issues, can you please open a ticket so our team can investigate further? Thanks!

@sjsmith haven't some for of change log would help us too for released kernels :)

I saw this today with 4.18.16-x86_64-linode118
Switched to using the upstream kernel to solve the problem.

I just got the same problem as @wbagdon. Does it mean I need to choose "GRUB 2" (at least for now)?

@thorner @sjsmith Is that a regression?

Same problem here, fixed by selecting GRUB 2

Another report of this issue with latest Linode kernel. 30 machines went down when running apt update

@gsgibson omg. I feel your pain. Thank god I'm not using Docker for any production stuff yet.

Here's an alternative work-around, provided by Linode support (if you really want to keep using the latest kernel):

As root or sudo:

$ mkdir -p /etc/systemd/system/containerd.service.d/
$ cat << EOF > /etc/systemd/system/containerd.service.d/override.conf [Service]
ExecStartPre=
EOF
$ systemctl restart docker

Linode kernel 4.18.16-x86_64-linode118 is missing overlay module.

So I found this thread.

Right now no time to check if it is already reported in an ticket.

Thing I want to report here is that the above workaround, reported by @k_sze, where containerd.service is overwritten to not do /sbin/modprobe overlay works for us.

Thanks.

Avatar esyoum 1 month ago Linode Staff

Docker 18.9.0 fix for latest Linode Kernel

Last week, Docker released a new version (18.9.0) that has been steadily failing against our latest Kernel. Our team spent a few days testing Docker 18.9.0 with Ubuntu 16.04 - 18.04 and Debian 9 in order to assist our customers with the best solution possible.

We found we could recreate the issue after installing Docker 18.9.0 according to the documentation.

Output from a fresh installation of Docker-CE on Ubuntu 16.04 includes the following error messages:

Job for containerd.service failed because the control process exited with error code.
See "systemctl status containerd.service" and "journalctl -xe" for details.
containerd.service couldn't start.
...
A dependency job for docker.service failed. See 'journalctl -xe' for details.
docker.service couldn't start.

The code above shows us that the containerd service is looking for the OverlayFS module and failing, as support for OverlayFS is compiled into our kernel, and has been since version 113. As a result we believe this to be a bug introduced by the upstream docker package maintainers.

In addition to the workaround to switch to the GRUB2 kernel, we found the following solution in a Github Docker issue discussion which doesnt require switching kernels.

To apply this fix, please follow the below steps:

  • Switch to root user
su -
  • Run the following commands (these have been updated from the commands provided by @k_sze):
mkdir -p /etc/systemd/system/containerd.service.d/

cat << EOF > /etc/systemd/system/containerd.service.d/override.conf
[Service]
ExecStartPre=
EOF

systemctl daemon-reload

systemctl start docker

When we began investigating this most recent Docker issue, we received many questions asking whether our kernels still do not have OverlayFS enabled. While this was an issue in the past, OverlayFS has since been enabled in our latest kernels. This is confirmed by the following outputs:

$ uname -r
4.18.16-x86_64-linode118

$ grep -i overlay /proc/filesystems
nodev   overlay

$ zcat /proc/config.gz  | grep -i overlay
CONFIG_OF_OVERLAY=y
CONFIG_OVERLAY_FS=y
CONFIG_OVERLAY_FS_REDIRECT_DIR=y
# CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW is not set
CONFIG_OVERLAY_FS_INDEX=y
# CONFIG_OVERLAY_FS_NFS_EXPORT is not set
CONFIG_OVERLAY_FS_XINO_AUTO=y

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct