Mysql database often stops and won't automatially restart

Server: Linode Dedicated 4G
Ubuntu 18.04, Webmin + Virtualmin

Every now and then, mysql server stops and fails on restarting. Here is the logs I was able to find:

Apr 11 20:54:57 bamboo kernel: [ 2082.915857] [ 7664]   112  7664   406072    46295   745472        3             0 mysqld
Apr 11 20:54:57 bamboo kernel: [ 2082.915859] [ 7701]     0  7701   129687      307   679936     1435             0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915860] [ 7702]     0  7702   129683      253   667648     1487             0 apache2
Apr 11 20:54:57 bamboo kernel: [ 2082.915861] Out of memory: Kill process 7664 (mysqld) score 40 or sacrifice child
Apr 11 20:54:57 bamboo kernel: [ 2082.919284] Killed process 7664 (mysqld) total-vm:1624288kB, anon-rss:185180kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo kernel: [ 2082.941447] oom_reaper: reaped process 7664 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Failed with result 'signal'.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 11 20:54:57 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 20:54:57 bamboo systemd[1]: Stopped MySQL Community Server.

..........

Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Scheduled restart job, restart counter is at 4.
Apr 11 14:02:53 bamboo systemd[1]: Stopped MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: Starting MySQL Community Server...
Apr 11 14:02:53 bamboo mysqld[4345]: Initialization of mysqld failed: 0
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Control process exited, code=exited status=1
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 11 14:02:53 bamboo systemd[1]: Failed to start MySQL Community Server.
Apr 11 14:02:53 bamboo systemd[1]: mysql.service: Service hold-off time over, scheduling restart.

Looks like it is a memory issue. But I am hosting some very small wordpress sites on this server, and it is impossible for them to use up the memory.

The server graphs are like https://imgur.com/a/Vtfc6Zf
<img alt="aaa" src="https://imgur.com/a/Vtfc6Zf">

You can see there is a CPU usage spike, and it is when the mysql databases got stopped.

I was not doing anything special at that time so it is quite hard for me to reproduce the problem and observer the possible cause. I guess to investigate this I need to look deeper into the logs, but this is totally beyond my knowledge.

Can someone help me with investigating this?

1 Reply

Looks like it is a memory issue. But I am hosting some very small wordpress sites on this server, and it is impossible for them to use up the memory.

The giveaway is this:

Apr 11 20:54:57 bamboo kernel: [ 2082.919284] Killed process 7664 (mysqld) total-vm:1624288kB, anon-rss:185180kB, file-rss:0kB, shmem-rss:0kB

It looks like you have some sort of attack in progress… I suggest you implement some countermeasures. Only you know what those should be…

As for restarting mysqld, you need to modify the systemd package that starts mysqld at boot up so that systemd will start mysqld when it dies. You should seek out a better advisor on this process than me.

-- sw

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct