Mysql database often stops and won't automatially restart
Server: Linode Dedicated 4G
Ubuntu 18.04, Webmin + Virtualmin
Every now and then, mysql server stops and fails on restarting. Here is the logs I was able to find:
Apr 11 20:54:57 bamboo kernel: [ 2082.915857] [ 7664] 112 7664 406072 46295 745472 3 0 mysqld
Apr 11 20:54:57 bamboo kernel: [ 2082.915859] [ 7701] 0 7701 129687 307 679936 1435 0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915860] [ 7702] 0 7702 129683 253 667648 1487 0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915861] Out of memory: Kill process 7664 (mysqld) score 40 or sacrifice child
Apr 11 20:54:57 bamboo kernel: [ 2082.919284] Killed process 7664 (mysqld) total-vm:1624288kB, anon-rss:185180kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo kernel: [ 2082.941447] oom_reaper: reaped process 7664 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Failed with result 'signal'.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 20:54:57 bamboo systemd[1]: Stopped MySQL Community Server.
..........
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 14:02:53 bamboo systemd[1]: Stopped MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: Starting MySQL Community Server...
Apr 11 14:02:53 bamboo mysqld[4345]: Initialization of mysqld failed: 0
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Control process exited, code=exited status=1
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 11 14:02:53 bamboo systemd[1]: Failed to start MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Looks like it is a memory issue. But I am hosting some very small wordpress sites on this server, and it is impossible for them to use up the memory.
The server graphs are like https://imgur.com/a/Vtfc6Zf
<img alt="aaa" src="https://imgur.com/a/Vtfc6Zf">
You can see there is a CPU usage spike, and it is when the mysql databases got stopped.
I was not doing anything special at that time so it is quite hard for me to reproduce the problem and observer the possible cause. I guess to investigate this I need to look deeper into the logs, but this is totally beyond my knowledge.
Can someone help me with investigating this?
1 Reply
Looks like it is a memory issue. But I am hosting some very small wordpress sites on this server, and it is impossible for them to use up the memory.
The giveaway is this:
Apr 11 20:54:57 bamboo kernel: [ 2082.919284] Killed process 7664 (mysqld) total-vm:1624288kB, anon-rss:185180kB, file-rss:0kB, shmem-rss:0kB
It looks like you have some sort of attack in progress… I suggest you implement some countermeasures. Only you know what those should be…
As for restarting mysqld, you need to modify the systemd package that starts mysqld at boot up so that systemd will start mysqld when it dies. You should seek out a better advisor on this process than me.
-- sw