PHP-FPM is failing after reboot with error 502 - why?
For some time now I am using Laravel Forge (ubuntu 16.04) on linode on a 1GB plan which has worked like a charm for months. Since I changed to get the free storage-upgrade last week, things changed…
After booting, I got an nginx error 502 in the browser. After checking logs, I quickly figured out that the php7.2-fpm.service was failing:
php7.2-fpm.service: Start operation timed out. Terminating.
php7.2-fpm.service: Failed with result 'timeout'.
Failed to start The PHP 7.2 FastCGI Process Manager.
SSH and tried to start the php7.2-fpm service manually:
sudo service php7.2-fpm start
… things work fine again. Until the next server-reboot, again php7.2-fpm is failing. I checked the log file of php but couldn't find any relevant error message. Curiously nothing changed in my server-configuration, as this issue started after the migration.
So in the end, I tried to provision a new 1gb linode, with a new, fresh forge (ubuntu 18.04) installation. Checked the installation, enabled a test-site, working. After rebooting… again the same issue, nginx error 502. Checking logs:
php7.2-fpm.service: Start operation timed out. Terminating.
php7.2-fpm.service: Failed with result 'timeout'.
Failed to start The PHP 7.2 FastCGI Process Manager.
I can however start this service again manually via SSH.
The only noticeable difference after the plan-upgrade is the following:
On the previous plan, my nanode was running on a Xeon V4 server (I forgot the prefix), now it's running on a noticeably slower Xeon E5-2680 v3 server. I'm not complaining on this, but something is causing PHP not longer starting, which is my main issue. If the issue happens as well on my existing droplet (which was running fine before), as well on a freshly installed droplet, I am slowly starting to guess it must be something else as it all started, when the server-hardware was changed…
Question:
- Any ideas and hints, where to start first?
- And most important: any workarounds how to get the php7.2-fpm service running after all after failing to start, even without the need to interact manually by myself through the terminal (in case of an unexpected server reboot)?
10 Replies
If you check using
service php7.2-fpm status
.. right after reboot, you will notice the reason is an exit code like the following :
â— php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; > vendor preset: enabled)
Active: failed (Result: timeout) since Wed 2018-08-22 15:38:28 UTC; 1min 12s ago
Docs: man:php-fpm7.2(8)
Process: 671 ExecStart=/usr/sbin/php-fpm7.2 --nodaemonize --fpm-config /etc/php/7.2/fpm/php-fpm.conf (code=killed, signal=TERM)
Main PID: 671 (code=killed, signal=TERM)Aug 22 15:36:58 localhost systemd[1]: Starting The PHP 7.2 FastCGI > Process Manager…
Aug 22 15:38:28 localhost systemd[1]: php7.2-fpm.service: Start operation timed out. Terminating.
Aug 22 15:38:28 localhost systemd[1]: Failed to start The PHP 7.2 FastCGI Process Manager.
Aug 22 15:38:28 localhost systemd[1]: php7.2-fpm.service: Unit entered failed state.
Aug 22 15:38:28 localhost systemd[1]: php7.2-fpm.service: Failed with result 'timeout'.
The problem is this :
Failed with result 'timeout'
Which does look like the slower CPU (perhaps during the "upgrade" it moved you to a box which is overloaded causing programs to generally run or start slower) is the core issue here.
I am having the same problem, where both apache2 and php7.2-fpm no longer start, however I can start them manually after everything else has initialized and I can shell into the box.
mwildman
Linode Staff
It's possible that this could be related to your Linode's kernel version, especially if your Linode is configured to boot using the latest kernel available. I've seen a few cases where rolling back to a previous release solved this, specifically rolling back to "4.15.13-x86_64-linode106". For your reference, here's a list of all the kernels that we have available:
Here's a guide on updating the kernel version of your Linode:
How to change your Linode's Kernel
Let us know how this works out for you.
I experimented with different kernels from the panel. The issue indeed starts from version 4.16.11-x86_64-linode108 and higher.
During migration, linode automatically updated to the latest 4.17 kernel on my forge webserver. That is probably where it went wrong.
Eventually I have chosen the option GRUB 2 to use the kernel from Ubuntu 18.04 itself (currently using 4.15.0.33-generic)
So great, that works all ok again, thanks a lot!
Last practical question:
Is there any specific reason to replace the generic kernel with Ubuntu 18.04 LTS ("4.15.0.33-generic") with a linode kernel instead? (for example "4.15.13-x86_64-linode106")
Was having this same issue, can confirm reverting back to "4.15.13-x86_64-linode106" kernel fixed it for me.
Thanks for this guys!
I’ve just migrated to Linode (4GB) and had exactly the same issue with PHP 7.2 FPM and snapd failing.
Using the standard Ubuntu kernel (GRUB 2, 4.15.0) instead of 4.17 fixed it for me also.
This is how I solved this problem during my 18.04 LTS install:
Add extended start and stop times due to long wait (~3 minutes), apparently for sodium initialize after boot.
sudo vi /etc/systemd/system/multi-user.target.wants/php7.2-fpm.service
TimeoutStartSec=300s
TimeoutStopSec=300s
I am currently using 4.17.15-x86_64-linode115.
I have a couple probably dumb questions related to this, as I too ran into this issue and was happy to find some answers related to it here.
I switched back to kernel 4.15.18…-linode 107 to fix this startup issue, and it indeed seems to work. Moving forward, are security fixes backported to this particular kernel, or do I need to look for new kernel releases from Linode in this 4.15.x tree and switch to them moving forward?
smoetje asks above, "Is there any specific reason to replace the generic kernel with Ubuntu 18.04 LTS ("4.15.0.33-generic") with a linode kernel instead? (for example "4.15.13-x86_64-linode106")," and I'm also curious about this. I always used the "latest 64 bit x" option, but now that there seems to be a bug in that, I'm a little unsure what the best possible kernel strategy should be. I don't do anything cutting edge with my vps… just php/mysql/mail really, but I always defaulted to the latest kernel simply so I wouldn't have to think about it with future upgrades. Does it make more sense to Grub 2 it up, or is there a tangible benefit to using the linode kernels? Perhaps some Linode goodness baked into these that would be lost by going Grub?
We had the same problem with a couple of Linode machines, but only the smallest setup: Nanode 1GB. The bigger Linodes are not having this problem with the newest kernel. All services are coming back after reboot.
However it would be interesting to know from which side this problem can be tackled. Maybe the forge provisioning script can be changed.
Is this something that can be fixed by Linode in one of its "latest" kernel releases, or does this need to be fixed by Debian/Ubuntu? Or maybe it's already fixed in a latest kernel release since this thread started?
Apparently installing haveged has fixed this issue for some people. From: https://github.com/jedisct1/libsodium-php/issues/94
Not sure why the newer kernels would have a problem while earlier ones do not, but maybe something to do with how they use random?