Docker won't start using the latest Linode kernel
I recently rebooted my Linode and Docker will no longer start. I have my Linode configured to boot with the latest 64bit kernel. Prior to rebooting, Docker was starting without error. How do I fix this?
Our team is aware of the issue with Docker and our recent kernels:
A specific module for OverlayFS was not present in these kernels which caused incompatibilities with Docker. This module is now included in our latest revisions,
There are 2 easy solutions to get Docker back up and running:
- Reboot your Linode to boot with the latest Linode kernel.
Currently the latest 32bit and 64bit Linode kernels include support for OverlayFS.
Make sure your Linode is configured to boot using our Latest kernel within your configuration profile. This selection will allow your Linode to boot using the latest stable Linode kernel at each reboot.
- Use your distribution's upstream kernel.
Since all Linodes run on KVM, they are able to boot using either a Linode supplied kernel or your distribution's upstream kernel. You may have noticed that the newest distributions are already configured to use GRUB2 and your distribution's upstream kernel. This is the now the default method for all new Images and the preferred selection for kernels. To boot using a distribution supplied kernel, follow the steps in our guide here:
Running a Distribution-Supplied Kernel
Why would you deliberately release (and make default) a kernel which breaks a widely used piece of software, without announcing it?
It also breaks networking on a Debian 7 server I manage.
@turnip our team released a new set of 4.17.14 kernels today (linode114 and linode133) which should resolve this for you. If you are still seeing any Docker or networking issues, can you please open a ticket so our team can investigate further? Thanks!
I saw this today with
Switched to using the upstream kernel to solve the problem.
Another report of this issue with latest Linode kernel. 30 machines went down when running apt update
Here's an alternative work-around, provided by Linode support (if you really want to keep using the latest kernel):
As root or sudo:
$ mkdir -p /etc/systemd/system/containerd.service.d/ $ cat << EOF > /etc/systemd/system/containerd.service.d/override.conf [Service] ExecStartPre= EOF $ systemctl restart docker
4.18.16-x86_64-linode118 is missing
So I found this thread.
Right now no time to check if it is already reported in an ticket.
Thing I want to report here is that the above workaround, reported by @k_sze, where
containerd.service is overwritten to not do
/sbin/modprobe overlay works for us.
Docker 18.9.0 fix for latest Linode Kernel
Last week, Docker released a new version (18.9.0) that has been steadily failing against our latest Kernel. Our team spent a few days testing Docker 18.9.0 with Ubuntu 16.04 - 18.04 and Debian 9 in order to assist our customers with the best solution possible.
We found we could recreate the issue after installing Docker 18.9.0 according to the documentation.
Output from a fresh installation of Docker-CE on Ubuntu 16.04 includes the following error messages:
Job for containerd.service failed because the control process exited with error code. See "systemctl status containerd.service" and "journalctl -xe" for details. containerd.service couldn't start. ... A dependency job for docker.service failed. See 'journalctl -xe' for details. docker.service couldn't start.
The code above shows us that the containerd service is looking for the OverlayFS module and failing, as support for OverlayFS is compiled into our kernel, and has been since version 113. As a result we believe this to be a bug introduced by the upstream docker package maintainers.
In addition to the workaround to switch to the GRUB2 kernel, we found the following solution in a Github Docker issue discussion which doesnt require switching kernels.
To apply this fix, please follow the below steps:
- Switch to root user
- Run the following commands (these have been updated from the commands provided by @k_sze):
mkdir -p /etc/systemd/system/containerd.service.d/ cat << EOF > /etc/systemd/system/containerd.service.d/override.conf [Service] ExecStartPre= EOF systemctl daemon-reload systemctl start docker
When we began investigating this most recent Docker issue, we received many questions asking whether our kernels still do not have OverlayFS enabled. While this was an issue in the past, OverlayFS has since been enabled in our latest kernels. This is confirmed by the following outputs:
$ uname -r 4.18.16-x86_64-linode118 $ grep -i overlay /proc/filesystems nodev overlay $ zcat /proc/config.gz | grep -i overlay CONFIG_OF_OVERLAY=y CONFIG_OVERLAY_FS=y CONFIG_OVERLAY_FS_REDIRECT_DIR=y # CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW is not set CONFIG_OVERLAY_FS_INDEX=y # CONFIG_OVERLAY_FS_NFS_EXPORT is not set CONFIG_OVERLAY_FS_XINO_AUTO=y