Daily my DB is crashing

Running out of memory?

It's a pretty common situation when you're running Apache and MySQL/MariaDB with default settings on a smallish Linode.

What is the solution for this ?

Why is it crashing?

Ok, If I increase it to 2GB plan, will this crash solve ?

You need to figure out -why- it is crashing. Simply increasing your node is not a solution. If it is crashing because it goes out of memory that means you configured it to use more memory than that you have available. Either change the config to use less memory or increase the node if you need stuff stored in memory. Though even then, if your settings are to let mysql consume 20GB memory (as an example) upgrading to 2GB obviously isn't a solution. So first fix the config file so you know how much memory MySQL can use.

mysqltuner is a nice utility that helps you figure this out. It'll tell you the maximum possible memory that MySQL can consume (http://mysqltuner.pl)

Hi,

Actually this is the log am getting from mysqltuner. Can you please suggest me the solution, Am using 1GB RAM Linode

http://www.codesend.com/view/a3c59e7c97 … fe840b564/">http://www.codesend.com/view/a3c59e7c975a4d464369e39fe840b564/

You have query/thread caching disabled. I'd enable them and lower the max connections from 151 to 50 or something. However as you can see here:

[OK] Maximum reached memory usage: 321.4M (32.41% of installed RAM)

[OK] Maximum possible memory usage: 708.0M (71.39% of installed RAM)

It seems that at the moment the memory usage is fine, unless your other applications are taking up all the RAM. You should check the logfiles to figure out why it is crashing,

Often, what happens is that Apache uses up all the RAM and triggers the kernel's OOM (out-of-memory) killer. But Apache can usually survive one of its children being killed, so it just respawns the killed child and keeps on using a ton of RAM. Sooner or later, the kernel realizes that it's futile to try to kill Apache, so it kills MySQL/MariaDB instead. The poor database did nothing wrong, it just got sacrificed to make room for greedy Apache.

If that's what's happening, tweaking the database isn't going to help. You have to tweak all the other major programs that are running on the same server. Apache is the most common culprit. If you installed it with an off-the-shelf control panel, it's unlikely to have been optimized for your 1GB instance.

Stop using Apache and switch to Nginx + PHP-FPM.

There's nothing wrong with using Apache if it's configured correctly.

Well, the problem with Apache is that there are a lot of different ways to configure it and not all of them are suitable for a Linode 1024.

The OOM killer will probably go away just by using Apache + PHP-FPM, so that would be a good start.

When I run # free -lm

# free -lm
              total        used        free      shared  buff/cache   available
Mem:            991         787          11           3         193         161
Low:            991         980          11
High:             0           0           0
Swap:           255         193          62

Its really frustrating for me. Daily my site will be offline for one hour. I need to keep tracking for all the time. Please suggest me any solutions for this.

Will this help, If I add this in my.cnf

 <ifmodule mpm_worker_module="">ServerLimit          40
    StartServers          2
    MaxClients          1000
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0</ifmodule> 

my.cnf is the configuration file for MySQL/MariaDB.

For Apache, it has to go somewhere in /etc/httpd or /etc/apache2, depending on which Linux distribution you're using.

You probably aren't using mpmworker, either. Most Apache setups that cause memory issues use mpmprefork.

Go to the directory I mentioned above and look in all the files to find the mpm_prefork section.

Set MaxClients (or MaxRequestWorkers, if it's there) to 10. Not 1000, not 100, just 10. Then restart Apache.

Yes I have updated the httpd.conf to

 <ifmodule mpm_prefork_module="">StartServers          5
    MinSpareServers       5
    MaxSpareServers     10
    MaxClients          10
    MaxRequestsPerChild   0</ifmodule> 

and this is my top -bn 1, am not sure why multiple apache is running and nearly 70% of mem is used even with no visitors are there. Please check this and give solutions.

top - 18:03:30 up 21 days, 18:52,  1 user,  load average: 0.28, 0.19, 0.33
Tasks: 132 total,   1 running, 130 sleeping,   1 stopped,   0 zombie
%Cpu(s):  3.2 us,  0.8 sy,  0.0 ni, 95.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.1 st
KiB Mem :  1015488 total,    11028 free,   750008 used,   254452 buff/cache
KiB Swap:   262140 total,        0 free,   262140 used.   184256 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    1 root      20   0  285244   4008   2308 S  0.0  0.4   4:06.39 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.23 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:23.91 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S  0.0  0.0   3:24.69 rcu_sched
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 khelper
   11 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   12 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 perf
  257 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
  259 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
  260 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
  261 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
  263 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
  344 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ata_sff
  358 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
  452 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 rpciod
  562 root      20   0       0      0      0 S  0.0  0.0  11:23.99 kswapd0
  637 root      20   0       0      0      0 S  0.0  0.0   0:00.00 fsnotify_mark
  663 root      20   0       0      0      0 S  0.0  0.0   0:00.00 ecryptfs-kthrea
  665 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 nfsiod
  673 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 cifsiod
  685 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jfsIO
  686 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jfsCommit
  687 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jfsSync
  693 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfsalloc
  694 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs_mru_cache
  696 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 glock_workqueue
  697 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 delete_workqueu
  703 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 gfs_recovery
  791 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 acpi_thermal_pm
 1357 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/u3:0
 1358 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kloopd
 1408 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
 1409 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 drbd-reissue
 1411 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 rbd
 1413 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
 1414 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
 1426 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
 1429 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
 1430 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_2
 1445 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_2
 1467 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 raid5wq
 1479 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dm_bufio_cache
 1502 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
 1517 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ceph-msgr
 1532 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
 1551 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
 1559 root       0 -20       0      0      0 S  0.0  0.0   1:34.34 kworker/0:1H
 1560 root      20   0       0      0      0 S  0.0  0.0   0:42.39 jbd2/sda-8
 1561 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
 1578 root      20   0  106760   4376   4168 S  0.0  0.4  12:38.29 systemd-journal
 1720 root      20   0       0      0      0 S  0.0  0.0   0:20.81 kauditd
 2578 root      20   0   41224      4      4 S  0.0  0.0   0:00.06 systemd-udevd
 2606 root      16  -4  116716    140      0 S  0.0  0.0   2:14.39 auditd
 2651 avahi     20   0   28084    120      0 S  0.0  0.0   0:16.07 avahi-daemon
 2655 chrony    20   0   26800    108      0 S  0.0  0.0   0:02.44 chronyd
 2658 root      20   0  446728   2260   1396 S  0.0  0.2   2:52.80 NetworkManager
 2660 root      20   0  250640   7652   7248 S  0.0  0.8   5:01.38 rsyslogd
 2661 avahi     20   0   27952      0      0 S  0.0  0.0   0:00.00 avahi-daemon
 2662 root      20   0  550156    372      0 S  0.0  0.0   5:21.41 tuned
 2665 dbus      20   0   26720    368     64 S  0.0  0.0   2:28.76 dbus-daemon
 2667 root      20   0   34692   2020   1876 S  0.0  0.2   1:30.66 systemd-logind
 2693 polkitd   20   0  513996    432      0 S  0.0  0.0   1:11.89 polkitd
 2969 root      20   0   82504   1192   1104 S  0.0  0.1   1:22.21 sshd
 2986 root      20   0  520652    164     84 S  0.0  0.0   2:46.49 php-fpm
 3015 root      20   0  201908   1164   1072 S  0.0  0.1   0:13.16 pure-ftpd
 3025 root      20   0  126312   1424   1324 S  0.0  0.1   0:23.27 crond
 3030 named     20   0  161360    124      0 S  0.0  0.0   0:00.70 named
 3037 root      20   0  110012      4      4 S  0.0  0.0   0:00.00 agetty
 3039 root      20   0  110012      4      4 S  0.0  0.0   0:00.01 agetty
 3189 apache    20   0  520652     12      0 S  0.0  0.0   0:00.00 php-fpm
 3190 apache    20   0  520652     12      0 S  0.0  0.0   0:00.00 php-fpm
 3191 apache    20   0  520652     12      0 S  0.0  0.0   0:00.00 php-fpm
 3192 apache    20   0  520652     12      0 S  0.0  0.0   0:00.00 php-fpm
 3197 apache    20   0  520652     12      0 S  0.0  0.0   0:00.00 php-fpm
 3209 root      20   0  211292   1004    608 S  0.0  0.1   0:10.92 fail2ban-server
 3309 root      20   0   91080    868    784 S  0.0  0.1   0:19.44 master
 3313 postfix   20   0   91344    164      0 S  0.0  0.0   0:02.48 qmgr
 3398 root      20   0   19848   1440   1300 S  0.0  0.1   0:23.78 dovecot
 3403 dovecot   20   0    9320     84      0 S  0.0  0.0   0:06.95 anvil
 3404 root      20   0    9684   1040    868 S  0.0  0.1   0:07.77 log
 3554 postfix   20   0   91196    104      0 S  0.0  0.0   0:02.10 tlsmgr
17875 postfix   20   0   91184   1148    132 S  0.0  0.1   0:00.00 pickup
17948 postfix   20   0  167688   3416   1980 S  0.0  0.3   0:00.03 smtpd
18377 postfix   20   0   91176   1008      0 S  0.0  0.1   0:00.00 anvil
18636 root      20   0  137768   1208      0 S  0.0  0.1   0:00.95 sshd
18640 root      20   0  115496   2836   2340 S  0.0  0.3   0:00.09 bash
19048 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kworker/0:3
19921 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/u2:0
20038 root      20   0       0      0      0 S  0.0  0.0   0:00.03 kworker/0:2
20229 root      20   0   16428    960     44 S  0.0  0.1   0:00.00 config
20249 amavis    20   0  468104 317324  10760 S  0.0 31.2   0:19.76 clamd
20564 root      20   0  120660   1732   1568 T  0.0  0.2   0:00.00 vmstat
20595 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0
20706 root      20   0  168368   3212   2772 S  0.0  0.3   0:00.00 crond
20713 root      20   0  113124   2272   2120 S  0.0  0.2   0:00.00 freshclam-sleep
20732 root      20   0  107900    292    216 S  0.0  0.0   0:00.00 sleep
20803 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/0:1
20806 root      20   0  615656  39580  28076 S  0.0  3.9   0:00.25 httpd
20809 root      20   0   29736   5348   3420 S  0.0  0.5   0:00.04 vlogger (access
20810 apache    20   0  267348   8972   5124 S  0.0  0.9   0:00.00 httpd
20812 apache    20   0  616188  25332  13280 S  0.0  2.5   0:00.02 httpd
20813 apache    20   0  616444  24024  11880 S  0.0  2.4   0:00.03 httpd
20814 apache    20   0  616340  25076  12916 S  0.0  2.5   0:00.01 httpd
20815 apache    20   0  615940  24592  12876 S  0.0  2.4   0:00.00 httpd
20816 apache    20   0  616340  24036  11900 S  0.0  2.4   0:00.00 httpd
20859 web4      20   0  497112  99820  26844 S  0.0  9.8   0:03.67 php-cgi
20866 apache    20   0  615932  24660  12892 S  0.0  2.4   0:00.00 httpd
20991 apache    20   0  616088  24828  12884 S  0.0  2.4   0:00.00 httpd
20992 web4      20   0  498348 100932  27012 S  0.0  9.9   0:04.22 php-cgi
21001 apache    20   0  616088  23756  11872 S  0.0  2.3   0:00.00 httpd
21002 web4      20   0  476744  86844  24000 S  0.0  8.6   0:01.02 php-cgi
21010 apache    20   0  615800  24352  12768 S  0.0  2.4   0:00.00 httpd
21011 apache    20   0  615800  24388  12780 S  0.0  2.4   0:00.00 httpd
21016 ispconf+  20   0  421580  29644  22460 S  0.0  2.9   0:00.05 php-cgi
21020 ispconf+  20   0  424344  20692  10180 S  0.0  2.0   0:00.17 php-cgi
21029 ispconf+  20   0  421580  29460  22272 S  0.0  2.9   0:00.06 php-cgi
21033 ispconf+  20   0  424592  20648  10124 S  0.0  2.0   0:00.17 php-cgi
21036 root      20   0  136708   7888   6568 S  0.0  0.8   0:00.04 sshd
21037 sshd      20   0   83928   5612   4668 S  0.0  0.6   0:00.00 sshd
21062 root      20   0  129892   3020   2580 R  0.0  0.3   0:00.00 top
23787 root      20   0   15452      0      0 S  0.0  0.0   0:00.00 ssl-params
28185 amavis    20   0  367516    628      0 S  0.0  0.1   0:01.90 /usr/sbin/amavi
28186 amavis    20   0  369384   4112      4 S  0.0  0.4   0:00.47 /usr/sbin/amavi
28187 amavis    20   0  370528  57840      4 S  0.0  5.7   0:00.94 /usr/sbin/amavi
29640 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kworker/u2:2
29701 mysql     20   0  115356      4      4 S  0.0  0.0   0:00.01 mysqld_safe
29951 mysql     20   0 1171588  22004      0 S  0.0  2.2   0:51.76 mysqld
30025 root      20   0  150720   1212   1000 S  0.0  0.1   0:00.19 auth

20249 amavis 20 0 468104 317324 10760 S 0.0 31.2 0:19.76 clamd

Aha! There's your culprit. Clamd (a virus scanner) is using over 30% of your RAM. That's not the kind of program that you want to run in a resource-constrained environment such as a 1GB linode. Especially if you don't get a lot of email through that linode.

On the other hand, the weird mix of php-fpm and php-cgi makes me wonder whether it was appropriate to change the mpm_prefork settings. Your server seems to be set up in a rather unusual way, possibly because of the control panel you chose.

I tried to kill amavis with port number, Its not working. Can you please suggest me how to disable or best open source control panel ?