Apache or MySQL crash

It's trying to free up RAM because you're exhausting your RAM. Figure out what is using the RAM, then tune it to use less RAM or upgrade your system to have more RAM.

Longview/munin/htop all have varying degrees of usefulness there.

• Les

Seems like RAM usage was normal when it happen. (timestamps of logs and longview are different)

![](http://www.seomandarin.com/wp-content/u … -mysql.jpg">http://www.seomandarin.com/wp-content/uploads/longview-mysql.jpg" />

See the line

Jun 23 04:09:18 www kernel: Out of memory: Kill process 2831 (mysqld) score 57 or sacrifice child

MySQL got killed because there is no enough spam, either upgrade ram or optimize these service

According to this thread http://www.webtrafficexchange.com/solve … uffer-pool">http://www.webtrafficexchange.com/solved-mysql-crash-fatal-error-cannot-allocate-memory-buffer-pool

which is very similar to my setup. I have set:

## Edit /etc/my.cnf, and add the following line under the [mysqld] heading.
[mysqld]
innodb_buffer_pool_size=64M

Think that will help in my situation?

> there is no enough spam No, he does not need more spam, he just needs more swap.

@sanvila:

No, he does not need more spam, he just needs more swap.

More swap is never the best answer. It's far slower to access than physical memory. Issues with running out of memory should be addressed, as many folks have said here, by increasing the memory allocated to the system to provide more headroom, or by optimizing the processes on the system consuming unnecessary resources.

By looking at the graphs, I think he was not really running out of memory.

Instead, the Out Of Memory killer (OOM killer) decided somehow that some process had to die.

Let me quote Andries Brouwer:

> An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.

Source: https://lwn.net/Articles/104185/

My suggestion to have more swap is not for it to be actually used (which we all know it's a lot slower than real RAM),

it's just something that may help to keep the OOM killer away.

Try building a kernel without swap on a Linode 1024 and see what happens. With enough swap, it works.

Does somebody know how to "optimize" gcc? I for sure would not even know where to start.

Do I need a Linode 2048 to build the kernel? Surely not if I can use a Linode 1024 with 2G of swap.

You can reduce overcommiting in MySQL or Apache by "optimizing" them, but there is not a general recipe for optimizing everything.

BTW: What I do to avoid OOM killer myself (in addition to have plenty of swap, 1G or 2G), is to have a file called /etc/sysctl.d/10-no-overcommit.conf containing this:

vm.overcommit_ratio = 100
vm.overcommit_memory = 2

(enable with "sysctl –system").

Then I will worry whenever I see that the system runs slow, not whenever OOM killer randomly decides to kill a random process following some strange algorithm, as if they were passengers in a plane in the story by Brouwer.

> My suggestion to have more swap is not for it to be actually used (which we all know it's a lot slower than real RAM),

it's just something that may help to keep the OOM killer away.

> No, he does not need more spam, he just needs more swap.

Linode's documentation on swap: ~~[https://www.linode.com/docs/troubleshooting/troubleshooting-memory-and-networking-issues" target="_blank">](https://www.linode.com/docs/troubleshoo … ing-issues">https://www.linode.com/docs/troubleshooting/troubleshooting-memory-and-networking-issues]( They recommend no more than 256mb swap

In this post they say 256swap for 1gb ram is good. More will slow down the server. https://forum.linode.com/viewtopic.php?p=40217

So setting vm.overcommitratio = 100 and vm.overcommitmemory = 2 can stop the OOM and doesn't slow down my server?

Thanks

@robertstam:

So setting vm.overcommitratio = 100 and vm.overcommitmemory = 2 can stop the OOM and doesn't slow down my server?
Not exactly. Do not ever think of settting vm.overcommit_memory = 2 (which is what really disables OOM killer) without understanding the consequences. In particular, do not ever think of disabling OOM killer without increasing swap first.

Moreover, don't do this or that because you read it in a forum like this one, and don't do something just because I tell you that it works for me. Investigate for yourself and draw your own conclusions.

The problem of overcommiting is complex and it should be understood without over-simplifications.

Recommended readings:

http://www.etalabs.net/overcommit.html

http://serverfault.com/questions/362589 … mit-memory">http://serverfault.com/questions/362589/effects-of-configuring-vm-overcommit-memory

http://www.hskupin.info/2010/06/17/how- … der-linux/">http://www.hskupin.info/2010/06/17/how-to-fix-the-oom-killer-crashe-under-linux/

BTW: More available swap by itself will not make your system to go slower.

In either case, it's actual usage of it what could make your system to go slower, but not its mere availability.

@robertstam:

my Linode has: 1gb ram, ubuntu 14.04 with 256mb swap size. My apache/php/mysql configuration is according the Linode tutorials for 1gb ram.
Hmm. If you refer to this page:

https://www.linode.com/docs/websites/ap … e-pangolin">https://www.linode.com/docs/websites/apache/apache-2-web-server-on-ubuntu-12-04-lts-precise-pangolin

where it's suggested to install libapache2-mod-php5, you should really consider php5-fpm instead. Memory usage with php5-fpm is predictable and almost flat.

Well, the OP clearly did not follow the instructions in Linode's Ubuntu 14.04 LAMP guide, since it suggests a MaxClients of 30.

 <ifmodule mpm_prefork_module="">StartServers 2                                                
MinSpareServers 6                                             
MaxSpareServers 12                                            
MaxClients 30                                                 
MaxRequestsPerChild 3000</ifmodule> 

The posted syslog shows 36 Apache processes running - reduce MaxClients and tune your MySQL settings and your memory problems will lessen.

@Vance:

Well, the OP clearly did not follow the instructions in Linode's Ubuntu 14.04 LAMP guide, since it suggests a MaxClients of 30.

I count 39. Although my settings already were:

 <ifmodule mpm_prefork_module="">StartServers          2
    MinSpareServers       6
    MaxSpareServers      12
    MaxClients          30
    MaxRequestsPerChild   3000</ifmodule> 

That is really odd. For mpmprefork, the maximum number of Apache processes should be the MaxClients value plus one (for the master process which controls all the others). If you run the following command as root, do you see mpmprefork_module listed?

apache2ctl -M

@Vance:

That is really odd. For mpm_prefork, the maximum number of Apache processes should be the MaxClients value plus one

Perhaps OP has some scripts that use the pcntl_fork() function in PHP.

That could be; I hadn't thought of that. Do any of the typical web applications do that?

Server has been running quite well for the last few weeks, until now :roll:

I have only made 1 change since the mysql crashes occured

which is:

## Edit /etc/my.cnf, and add the following line under the [mysqld] heading.
[mysqld]
innodb_buffer_pool_size=64M

~~http://www.seomandarin.com/wp-content/u … crash2.jpg">http://www.seomandarin.com/wp-content/uploads/mysql-crash2.jpg" />

Mysql error log:

[size]150711 18:16:41 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150711 18:16:41 [Note] Plugin 'FEDERATED' is disabled.
150711 18:16:41 InnoDB: The InnoDB memory heap is disabled
150711 18:16:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150711 18:16:41 InnoDB: Compressed tables use zlib 1.2.8
150711 18:16:41 InnoDB: Using Linux native AIO
150711 18:16:42 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(67993600 bytes) failed; errno 12
150711 18:16:42 InnoDB: Completed initialization of buffer pool
150711 18:16:42 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150711 18:16:42 [ERROR] Plugin 'InnoDB' init function returned error.
150711 18:16:42 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150711 18:16:42 [ERROR] Unknown/unsupported storage engine: InnoDB
150711 18:16:42 [ERROR] Aborting

150711 18:16:42 [Note] /usr/sbin/mysqld: Shutdown complete

150711 18:16:43 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150711 18:16:43 [Note] Plugin 'FEDERATED' is disabled.
150711 18:16:43 InnoDB: The InnoDB memory heap is disabled
150711 18:16:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150711 18:16:43 InnoDB: Compressed tables use zlib 1.2.8
150711 18:16:43 InnoDB: Using Linux native AIO
150711 18:16:43 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(67993600 bytes) failed; errno 12
150711 18:16:43 InnoDB: Completed initialization of buffer pool
150711 18:16:43 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150711 18:16:43 [ERROR] Plugin 'InnoDB' init function returned error.
150711 18:16:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150711 18:16:43 [ERROR] Unknown/unsupported storage engine: InnoDB
150711 18:16:43 [ERROR] Aborting

150711 18:16:43 [Note] /usr/sbin/mysqld: Shutdown complete[/size]

Syslog:

[size]Jul 11 18:16:34 www kernel: apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Jul 11 18:16:34 www kernel: apache2 cpuset=/ mems_allowed=0
Jul 11 18:16:34 www kernel: CPU: 0 PID: 11467 Comm: apache2 Not tainted 4.1.0-x86_64-linode59 #1
Jul 11 18:16:34 www kernel: 0000000000000000 0000000000000000 ffffffff8195b396 ffff88003cc0b180
Jul 11 18:16:34 www kernel: ffffffff81955e58 ffffffff811396e5 ffff88003e033400 ffff88003e033400
Jul 11 18:16:34 www kernel: ffffffff811b549f ffff88000257b9e8 01ff88000257b9e8 ffffea0000659e80
Jul 11 18:16:34 www kernel: Call Trace:
Jul 11 18:16:34 www kernel: [<ffffffff8195b396>] ? dump_stack+0x40/0x50
Jul 11 18:16:34 www kernel: [<ffffffff81955e58>] ? dump_header+0x7b/0x1fe
Jul 11 18:16:34 www kernel: [<ffffffff811396e5>] ? css_next_descendant_pre+0x1c/0x34
Jul 11 18:16:34 www kernel: [<ffffffff811b549f>] ? mem_cgroup_iter+0x117/0x319
Jul 11 18:16:34 www kernel: [<ffffffff811b8922>] ? vmpressure+0x1e/0x78
Jul 11 18:16:34 www kernel: [<ffffffff81961c55>] ? _raw_spin_unlock_irqrestore+0x2d/0x3e
Jul 11 18:16:34 www kernel: [<ffffffff81178dee>] ? oom_kill_process+0xc5/0x387
Jul 11 18:16:34 www kernel: [<ffffffff811789d6>] ? find_lock_task_mm+0x2c/0x7b
Jul 11 18:16:34 www kernel: [<ffffffff81179535>] ? __out_of_memory+0x433/0x473
Jul 11 18:16:34 www kernel: [<ffffffff811796b6>] ? out_of_memory+0x52/0x67
Jul 11 18:16:34 www kernel: [<ffffffff8117d811>] ? __alloc_pages_nodemask+0x724/0x862
Jul 11 18:16:34 www kernel: [<ffffffff8195f1d9>] ? yield+0x37/0x37
Jul 11 18:16:34 www kernel: [<ffffffff811ac7f6>] ? alloc_pages_vma+0x111/0x15a
Jul 11 18:16:34 www kernel: [<ffffffff811a6bf4>] ? read_swap_cache_async+0x76/0x121
Jul 11 18:16:34 www kernel: [<ffffffff811a6de4>] ? swapin_readahead+0x145/0x155
Jul 11 18:16:34 www kernel: [<ffffffff81175f91>] ? find_get_entry+0x1a/0x68
Jul 11 18:16:34 www kernel: [<ffffffff81176aa3>] ? pagecache_get_page+0x27/0x161
Jul 11 18:16:34 www kernel: [<ffffffff8119a6de>] ? handle_mm_fault+0xa88/0xf00
Jul 11 18:16:34 www kernel: [<ffffffff81003415>] ? xen_load_sp0+0xe0/0x12c
Jul 11 18:16:34 www kernel: [<ffffffff810450e4>] ? __do_page_fault+0x320/0x37b
Jul 11 18:16:34 www kernel: [<ffffffff81963e58>] ? page_fault+0x28/0x30
Jul 11 18:16:34 www kernel: Mem-Info:
Jul 11 18:16:34 www kernel: active_anon:110494 inactive_anon:110526 isolated_anon:0
Jul 11 18:16:34 www kernel: active_file:604 inactive_file:653 isolated_file:0
Jul 11 18:16:34 www kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Jul 11 18:16:34 www kernel: slab_reclaimable:6340 slab_unreclaimable:4953
Jul 11 18:16:34 www kernel: mapped:3819 shmem:3968 pagetables:2184 bounce:0
Jul 11 18:16:34 www kernel: free:2024 free_pcp:37 free_cma:0
Jul 11 18:16:34 www kernel: Node 0 DMA free:3948kB min:60kB low:72kB high:88kB active_anon:4472kB inactive_anon:4524kB active_file:32kB inactive_file:72kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:464kB slab_reclaimable:212kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:720 all_unreclaimable? yes
Jul 11 18:16:34 www kernel: lowmem_reserve[]: 0 970 970 970
Jul 11 18:16:34 www kernel: Node 0 DMA32 free:4148kB min:3952kB low:4940kB high:5928kB active_anon:437504kB inactive_anon:437580kB active_file:2384kB inactive_file:2540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032192kB managed:995552kB mlocked:0kB dirty:0kB writeback:0kB mapped:15244kB shmem:15408kB slab_reclaimable:25148kB slab_unreclaimable:19528kB kernel_stack:2560kB pagetables:8656kB unstable:0kB bounce:0kB free_pcp:148kB local_pcp:148kB free_cma:0kB writeback_tmp:0kB pages_scanned:68800 all_unreclaimable? yes
Jul 11 18:16:34 www kernel: lowmem_reserve[]: 0 0 0 0
Jul 11 18:16:34 www kernel: Node 0 DMA: 5*4kB (UR) 1*8kB (R) 1*16kB (R) 2*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 3948kB
Jul 11 18:16:34 www kernel: Node 0 DMA32: 215*4kB (UE) 31*8kB (UER) 18*16kB (R) 10*32kB (R) 8*64kB (R) 5*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 4148kB
Jul 11 18:16:34 www kernel: 6057 total pagecache pages
Jul 11 18:16:34 www kernel: 831 pages in swap cache
Jul 11 18:16:34 www kernel: Swap cache stats: add 79452, delete 78621, find 71219/75780
Jul 11 18:16:34 www kernel: Free swap  = 0kB
Jul 11 18:16:34 www kernel: Total swap = 262140kB
Jul 11 18:16:34 www kernel: 262047 pages RAM
Jul 11 18:16:34 www kernel: 0 pages HighMem/MovableOnly
Jul 11 18:16:34 www kernel: 9181 pages reserved
Jul 11 18:16:34 www kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jul 11 18:16:34 www kernel: [ 1561]     0  1561      919        3       5       2      207             0 upstart-udev-br
Jul 11 18:16:34 www kernel: [ 1567]     0  1567     2505        1       6       2      119         -1000 systemd-udevd
Jul 11 18:16:34 www kernel: [ 2523]     0  2523     1379       39       7       2      396             0 dhclient
Jul 11 18:16:34 www kernel: [ 2705]   105  2705     1060        1       6       2       79             0 dbus-daemon
Jul 11 18:16:34 www kernel: [ 2754]     0  2754     1055        1       5       2       71             0 systemd-logind
Jul 11 18:16:34 www kernel: [ 2756]   101  2756     7686       57       9       2      106             0 rsyslogd
Jul 11 18:16:34 www kernel: [ 2784]     0  2784     1951       12       8       2      111         -1000 sshd
Jul 11 18:16:34 www kernel: [ 2791]     0  2791      765       10       5       2       41             0 cron
Jul 11 18:16:34 www kernel: [ 2798]     0  2798      851        3       5       2      190             0 upstart-socket-
Jul 11 18:16:34 www kernel: [ 2802]     0  2802      756       17       5       2       78             0 upstart-file-br
Jul 11 18:16:34 www kernel: [ 2842]   104  2842    66538     5804      62       2     8543             0 mysqld
Jul 11 18:16:34 www kernel: [ 2869]     0  2869     8717     2055      17       2     1632             0 linode-longview
Jul 11 18:16:34 www kernel: [ 2991]     0  2991     9667      613      14       2      521             0 fail2ban-server
Jul 11 18:16:34 www kernel: [ 3035]     0  3035      606        1       5       2       28             0 getty
Jul 11 18:16:34 www kernel: [ 4034]     0  4034     1077       20       7       2       48             0 ntpd
Jul 11 18:16:34 www kernel: [ 4035]   103  4035     1046       14       7       2       42             0 ntpd
Jul 11 18:16:34 www kernel: [ 5878]     0  5878    31138      390      33       2      856             0 apache2
Jul 11 18:16:34 www kernel: [ 9188]    33  9188    43337    12691      85       2     3853             0 apache2
Jul 11 18:16:34 www kernel: [ 9214]    33  9214    41373    11431      82       2     3127             0 apache2
Jul 11 18:16:34 www kernel: [ 9241]    33  9241    39876     8462      79       2     4436             0 apache2
Jul 11 18:16:34 www kernel: [10485]    33 10485    39922    12156      78       2      733             0 apache2
Jul 11 18:16:34 www kernel: [10486]    33 10486    39721    12341      77       2      602             0 apache2
Jul 11 18:16:34 www kernel: [10657]    33 10657    39698     9866      77       2     2706             0 apache2
Jul 11 18:16:34 www kernel: [10914]    33 10914    39717     8909      77       2     3870             0 apache2
Jul 11 18:16:34 www kernel: [11181]    33 11181    39746     8494      78       2     3959             0 apache2
Jul 11 18:16:34 www kernel: [11192]    33 11192    39328     7451      77       2     4376             0 apache2
Jul 11 18:16:34 www kernel: [11249]    33 11249    39631     9259      77       2     3295             0 apache2
Jul 11 18:16:34 www kernel: [11462]    33 11462    38973     5740      46       2     3184             0 apache2
Jul 11 18:16:34 www kernel: [11466]    33 11466    38962     7115      46       2     2149             0 apache2
Jul 11 18:16:34 www kernel: [11467]    33 11467    38971     6980      46       2     2051             0 apache2
Jul 11 18:16:34 www kernel: [11470]    33 11470    38318     6764      45       2     1657             0 apache2
Jul 11 18:16:34 www kernel: [11471]    33 11471    38715     7221      46       2     1651             0 apache2
Jul 11 18:16:34 www kernel: [11472]    33 11472    38717     7173      46       2     1630             0 apache2
Jul 11 18:16:34 www kernel: [11473]    33 11473    38777     7250      46       2     1595             0 apache2
Jul 11 18:16:34 www kernel: [11478]    33 11478    38035     7179      45       2     1006             0 apache2
Jul 11 18:16:34 www kernel: [11479]    33 11479    38311     7930      45       2      993             0 apache2
Jul 11 18:16:34 www kernel: [11480]    33 11480    38311     7349      45       2     1025             0 apache2
Jul 11 18:16:34 www kernel: [11481]    33 11481    38311     7400      45       2      993             0 apache2
Jul 11 18:16:34 www kernel: [11482]    33 11482    38247     7619      45       2     1000             0 apache2
Jul 11 18:16:34 www kernel: [11483]    33 11483    38246     7436      45       2      964             0 apache2
Jul 11 18:16:34 www kernel: [11484]    33 11484    38247     7374      45       2      973             0 apache2
Jul 11 18:16:34 www kernel: [11485]    33 11485    38321     7478      45       2      917             0 apache2
Jul 11 18:16:34 www kernel: [11491]    33 11491    38247     7704      45       2      626             0 apache2
Jul 11 18:16:34 www kernel: [11492]    33 11492    38182     7678      45       2      659             0 apache2
Jul 11 18:16:34 www kernel: [11494]    33 11494    38247     7661      45       2      677             0 apache2
Jul 11 18:16:34 www kernel: [11496]    33 11496    38313     7714      45       2      642             0 apache2
Jul 11 18:16:34 www kernel: [11497]    33 11497    37860     7263      44       2      662             0 apache2
Jul 11 18:16:34 www kernel: [11499]    33 11499    31146      370      28       2      878             0 apache2
Jul 11 18:16:34 www kernel: [11501]    33 11501    31146      365      28       2      883             0 apache2
Jul 11 18:16:34 www kernel: [11502]    33 11502    31146      370      28       2      878             0 apache2
Jul 11 18:16:34 www kernel: [11503]    33 11503    31146      398      28       2      850             0 apache2
Jul 11 18:16:34 www kernel: [11504]    33 11504    31146      393      28       2      855             0 apache2
Jul 11 18:16:34 www kernel: [11505]    33 11505    31146      398      28       2      850             0 apache2
Jul 11 18:16:34 www kernel: [11506]    33 11506    31146      398      28       2      850             0 apache2
Jul 11 18:16:34 www kernel: [11507]    33 11507    31146      398      28       2      850             0 apache2
Jul 11 18:16:34 www kernel: [11508]    33 11508    31146      398      28       2      850             0 apache2
Jul 11 18:16:34 www kernel: [11509]    33 11509    31146      389      28       2      859             0 apache2
Jul 11 18:16:34 www kernel: Out of memory: Kill process 9188 (apache2) score 52 or sacrifice child
Jul 11 18:16:34 www kernel: Killed process 9188 (apache2) total-vm:173348kB, anon-rss:38844kB, file-rss:11920kB
Jul 11 18:16:40 www kernel: apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jul 11 18:16:40 www kernel: apache2 cpuset=/ mems_allowed=0
Jul 11 18:16:40 www kernel: CPU: 0 PID: 11473 Comm: apache2 Not tainted 4.1.0-x86_64-linode59 #1
Jul 11 18:16:40 www kernel: 0000000000000000 0000000000000000 ffffffff8195b396 ffff88003d7339c0
Jul 11 18:16:40 www kernel: ffffffff81955e58 ffffffff811396e5 ffff88003e033400 ffff88003e033400
Jul 11 18:16:40 www kernel: ffffffff811b549f ffff88000268ba78 01ff88000268ba78 ffffffff81004627
Jul 11 18:16:40 www kernel: Call Trace:
Jul 11 18:16:40 www kernel: [<ffffffff8195b396>] ? dump_stack+0x40/0x50
Jul 11 18:16:40 www kernel: [<ffffffff81955e58>] ? dump_header+0x7b/0x1fe
Jul 11 18:16:40 www kernel: [<ffffffff811396e5>] ? css_next_descendant_pre+0x1c/0x34
Jul 11 18:16:40 www kernel: [<ffffffff811b549f>] ? mem_cgroup_iter+0x117/0x319
Jul 11 18:16:40 www kernel: [<ffffffff81004627>] ? xen_mc_flush+0x11f/0x162
Jul 11 18:16:40 www kernel: [<ffffffff811b8922>] ? vmpressure+0x1e/0x78
Jul 11 18:16:40 www kernel: [<ffffffff81961c55>] ? _raw_spin_unlock_irqrestore+0x2d/0x3e
Jul 11 18:16:40 www kernel: [<ffffffff81178dee>] ? oom_kill_process+0xc5/0x387
Jul 11 18:16:40 www kernel: [<ffffffff811789d6>] ? find_lock_task_mm+0x2c/0x7b
Jul 11 18:16:40 www kernel: [<ffffffff81179535>] ? __out_of_memory+0x433/0x473
Jul 11 18:16:40 www kernel: [<ffffffff811796b6>] ? out_of_memory+0x52/0x67
Jul 11 18:16:40 www kernel: [<ffffffff8117d811>] ? __alloc_pages_nodemask+0x724/0x862
Jul 11 18:16:40 www kernel: [<ffffffff811aba9d>] ? alloc_pages_current+0xb2/0xcf
Jul 11 18:16:40 www kernel: [<ffffffff81177a30>] ? generic_file_read_iter+0x3ec/0x504
Jul 11 18:16:40 www kernel: [<ffffffff8119a9da>] ? handle_mm_fault+0xd84/0xf00
Jul 11 18:16:40 www kernel: [<ffffffff811c8886>] ? do_filp_open+0x30/0x74
Jul 11 18:16:40 www kernel: [<ffffffff811bae9e>] ? __vfs_read+0x93/0xb6
Jul 11 18:16:40 www kernel: [<ffffffff811bbd50>] ? vfs_read+0x85/0x11c
Jul 11 18:16:40 www kernel: [<ffffffff811bbe2d>] ? SyS_read+0x46/0x89
Jul 11 18:16:40 www kernel: [<ffffffff81964595>] ? sysenter_dispatch+0x7/0x21
Jul 11 18:16:40 www kernel: [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20
Jul 11 18:16:40 www kernel: [<ffffffff81008d25>] ? xen_force_evtchn_callback+0x9/0xa
Jul 11 18:16:40 www kernel: [<ffffffff81009312>] ? check_events+0x12/0x20
Jul 11 18:16:40 www kernel: Mem-Info:
Jul 11 18:16:40 www kernel: active_anon:110403 inactive_anon:110420 isolated_anon:32
Jul 11 18:16:40 www kernel: active_file:844 inactive_file:635 isolated_file:32
Jul 11 18:16:40 www kernel: unevictable:0 dirty:0 writeback:4 unstable:0
Jul 11 18:16:40 www kernel: slab_reclaimable:6328 slab_unreclaimable:4922
Jul 11 18:16:40 www kernel: mapped:842 shmem:3884 pagetables:2121 bounce:0
Jul 11 18:16:40 www kernel: free:1999 free_pcp:73 free_cma:0
Jul 11 18:16:40 www kernel: Node 0 DMA free:3952kB min:60kB low:72kB high:88kB active_anon:4372kB inactive_anon:4412kB active_file:128kB inactive_file:164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:464kB slab_reclaimable:212kB slab_unreclaimable:284kB kernel_stack:64kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1780 all_unreclaimable? yes
Jul 11 18:16:40 www kernel: lowmem_reserve[]: 0 970 970 970
Jul 11 18:16:40 www kernel: Node 0 DMA32 free:4044kB min:3952kB low:4940kB high:5928kB active_anon:437240kB inactive_anon:437268kB active_file:3248kB inactive_file:2376kB unevictable:0kB isolated(anon):128kB isolated(file):128kB present:1032192kB managed:995552kB mlocked:0kB dirty:0kB writeback:16kB mapped:3344kB shmem:15072kB slab_reclaimable:25100kB slab_unreclaimable:19404kB kernel_stack:2544kB pagetables:8404kB unstable:0kB bounce:0kB free_pcp:292kB local_pcp:292kB free_cma:0kB writeback_tmp:0kB pages_scanned:41164 all_unreclaimable? yes
Jul 11 18:16:40 www kernel: lowmem_reserve[]: 0 0 0 0
Jul 11 18:16:40 www kernel: Node 0 DMA: 6*4kB (UR) 1*8kB (R) 1*16kB (R) 2*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 3952kB
Jul 11 18:16:40 www kernel: Node 0 DMA32: 183*4kB (UEM) 34*8kB (MR) 18*16kB (R) 10*32kB (R) 8*64kB (R) 5*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 4044kB
Jul 11 18:16:40 www kernel: 6164 total pagecache pages
Jul 11 18:16:40 www kernel: 786 pages in swap cache
Jul 11 18:16:40 www kernel: Swap cache stats: add 91607, delete 90821, find 75334/83696
Jul 11 18:16:40 www kernel: Free swap  = 0kB
Jul 11 18:16:40 www kernel: Total swap = 262140kB
Jul 11 18:16:40 www kernel: 262047 pages RAM
Jul 11 18:16:40 www kernel: 0 pages HighMem/MovableOnly
Jul 11 18:16:40 www kernel: 9181 pages reserved
Jul 11 18:16:40 www kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jul 11 18:16:40 www kernel: [ 1561]     0  1561      919        3       5       2      207             0 upstart-udev-br
Jul 11 18:16:40 www kernel: [ 1567]     0  1567     2505        1       6       2      119         -1000 systemd-udevd
Jul 11 18:16:40 www kernel: [ 2523]     0  2523     1379       34       7       2      401             0 dhclient
Jul 11 18:16:40 www kernel: [ 2705]   105  2705     1060        1       6       2       79             0 dbus-daemon
Jul 11 18:16:40 www kernel: [ 2754]     0  2754     1055        1       5       2       71             0 systemd-logind
Jul 11 18:16:40 www kernel: [ 2756]   101  2756     7686       95       9       2      109             0 rsyslogd
Jul 11 18:16:40 www kernel: [ 2784]     0  2784     1951       12       8       2      111         -1000 sshd
Jul 11 18:16:40 www kernel: [ 2791]     0  2791      765       10       5       2       41             0 cron
Jul 11 18:16:40 www kernel: [ 2798]     0  2798      851        3       5       2      190             0 upstart-socket-
Jul 11 18:16:40 www kernel: [ 2802]     0  2802      756       17       5       2       78             0 upstart-file-br
Jul 11 18:16:40 www kernel: [ 2842]   104  2842    66538     5343      62       2     9024             0 mysqld
Jul 11 18:16:40 www kernel: [ 2869]     0  2869     8717     1723      17       2     1964             0 linode-longview
Jul 11 18:16:40 www kernel: [ 2991]     0  2991     9667      558      14       2      576             0 fail2ban-server
Jul 11 18:16:40 www kernel: [ 3035]     0  3035      606        1       5       2       28             0 getty
Jul 11 18:16:40 www kernel: [ 4034]     0  4034     1077       19       7       2       49             0 ntpd
Jul 11 18:16:40 www kernel: [ 4035]   103  4035     1046       13       7       2       43             0 ntpd
Jul 11 18:16:40 www kernel: [ 5878]     0  5878    31138      341      33       2      905             0 apache2
Jul 11 18:16:40 www kernel: [ 9214]    33  9214    41373     5153      82       2     6443             0 apache2
Jul 11 18:16:40 www kernel: [ 9241]    33  9241    40004     6056      79       2     4225             0 apache2
Jul 11 18:16:40 www kernel: [10485]    33 10485    39986     9280      78       2      855             0 apache2
Jul 11 18:16:40 www kernel: [10486]    33 10486    39945     9139      78       2      846             0 apache2
Jul 11 18:16:40 www kernel: [10657]    33 10657    39922     7515      78       2     2495             0 apache2
Jul 11 18:16:40 www kernel: [10914]    33 10914    39941     6502      78       2     3660             0 apache2
Jul 11 18:16:40 www kernel: [11181]    33 11181    39938     6295      79       2     3728             0 apache2
Jul 11 18:16:40 www kernel: [11192]    33 11192    39440     5665      78       2     4154             0 apache2
Jul 11 18:16:40 www kernel: [11249]    33 11249    39855     6949      77       2     3174             0 apache2
Jul 11 18:16:40 www kernel: [11462]    33 11462    38973     5826      46       2     3213             0 apache2
Jul 11 18:16:40 www kernel: [11466]    33 11466    39158     7350      47       2     2051             0 apache2
Jul 11 18:16:40 www kernel: [11467]    33 11467    39163     7233      47       2     2136             0 apache2
Jul 11 18:16:40 www kernel: [11470]    33 11470    38318     6587      45       2     1833             0 apache2
Jul 11 18:16:40 www kernel: [11471]    33 11471    39042     7675      46       2     1548             0 apache2
Jul 11 18:16:40 www kernel: [11472]    33 11472    39106     7727      46       2     1555             0 apache2
Jul 11 18:16:40 www kernel: [11473]    33 11473    39099     7636      46       2     1566             0 apache2
Jul 11 18:16:40 www kernel: [11478]    33 11478    38971     7946      46       2     1069             0 apache2
Jul 11 18:16:40 www kernel: [11479]    33 11479    39101     8272      46       2      999             0 apache2
Jul 11 18:16:40 www kernel: [11480]    33 11480    39038     8082      46       2     1072             0 apache2
Jul 11 18:16:40 www kernel: [11481]    33 11481    39034     8113      46       2      962             0 apache2
Jul 11 18:16:40 www kernel: [11482]    33 11482    39042     8098      46       2     1059             0 apache2
Jul 11 18:16:40 www kernel: [11483]    33 11483    38972     8167      46       2     1019             0 apache2
Jul 11 18:16:40 www kernel: [11484]    33 11484    38907     8017      46       2     1081             0 apache2
Jul 11 18:16:40 www kernel: [11485]    33 11485    39035     8330      46       2      926             0 apache2
Jul 11 18:16:40 www kernel: [11491]    33 11491    39034     8404      46       2      832             0 apache2
Jul 11 18:16:40 www kernel: [11492]    33 11492    38958     8291      46       2      775             0 apache2
Jul 11 18:16:40 www kernel: [11494]    33 11494    38976     8434      46       2      836             0 apache2
Jul 11 18:16:40 www kernel: [11496]    33 11496    39110     8993      46       2      878             0 apache2
Jul 11 18:16:40 www kernel: [11497]    33 11497    38912     8377      46       2      874             0 apache2
Jul 11 18:16:40 www kernel: [11499]    33 11499    31162      370      29       2      891             0 apache2
Jul 11 18:16:40 www kernel: [11501]    33 11501    31146      323      28       2      925             0 apache2
Jul 11 18:16:40 www kernel: [11502]    33 11502    31146      328      28       2      920             0 apache2
Jul 11 18:16:40 www kernel: [11503]    33 11503    31146      356      28       2      892             0 apache2
Jul 11 18:16:40 www kernel: [11504]    33 11504    31146      351      28       2      897             0 apache2
Jul 11 18:16:40 www kernel: [11505]    33 11505    31146      356      28       2      892             0 apache2
Jul 11 18:16:40 www kernel: [11506]    33 11506    31146      356      28       2      892             0 apache2
Jul 11 18:16:40 www kernel: [11507]    33 11507    31146      356      28       2      892             0 apache2
Jul 11 18:16:40 www kernel: [11508]    33 11508    31146      356      28       2      892             0 apache2
Jul 11 18:16:40 www kernel: [11509]    33 11509    31146      347      28       2      901             0 apache2
Jul 11 18:16:40 www kernel: Out of memory: Kill process 2842 (mysqld) score 45 or sacrifice child
Jul 11 18:16:40 www kernel: Killed process 2842 (mysqld) total-vm:266152kB, anon-rss:21372kB, file-rss:0kB
Jul 11 18:16:40 www kernel: init: mysql main process (2842) killed by KILL signal
Jul 11 18:16:40 www kernel: init: mysql main process ended, respawning
Jul 11 18:16:42 www kernel: init: mysql main process (11535) terminated with status 1
Jul 11 18:16:42 www kernel: init: mysql main process ended, respawning
Jul 11 18:16:43 www kernel: init: mysql post-start process (11536) terminated with status 1
Jul 11 18:16:43 www kernel: init: mysql main process (11570) terminated with status 1
Jul 11 18:16:43 www kernel: init: mysql respawning too fast, stopped[/size]</ffffffff81009312></ffffffff81008d25></ffffffff8100122a></ffffffff81964595></ffffffff811bbe2d></ffffffff811bbd50></ffffffff811bae9e></ffffffff811c8886></ffffffff8119a9da></ffffffff81177a30></ffffffff811aba9d></ffffffff8117d811></ffffffff811796b6></ffffffff81179535></ffffffff811789d6></ffffffff81178dee></ffffffff81961c55></ffffffff811b8922></ffffffff81004627></ffffffff811b549f></ffffffff811396e5></ffffffff81955e58></ffffffff8195b396></ffffffff81963e58></ffffffff810450e4></ffffffff81003415></ffffffff8119a6de></ffffffff81176aa3></ffffffff81175f91></ffffffff811a6de4></ffffffff811a6bf4></ffffffff811ac7f6></ffffffff8195f1d9></ffffffff8117d811></ffffffff811796b6></ffffffff81179535></ffffffff811789d6></ffffffff81178dee></ffffffff81961c55></ffffffff811b8922></ffffffff811b549f></ffffffff811396e5></ffffffff81955e58></ffffffff8195b396>

~~

You're still running out of memory, and the kernel is terminating random processes (usually apache2 or mysql) in order to reclaim memory.

Don't trust the memory graphs in Longview too much. Those graphs only update once every few minutes, so a very short spike in memory usage (e.g. caused by somebody opening your website in several browser windows at the same time) might not show up at all. Trust the kernel instead. If the kernel says you're running out of memory, you're running out of memory.

My suspicion is on Apache. You have over 30 apache2 processes, each consuming a copious amount of memory. You've tweaked MySQL since the last time, but you don't seem to have tweaked Apache at all. It's time to take a long hard look at what Apache is doing.

I know that the documentation recommends 30 processes for a 1GB linode. But the documentation might not be appropriate for your usage pattern, because every website is different. 30 processes are okay if you're only serving static files or running simple PHP scripts. If you're using anything serious, such as WordPress, Drupal, or any custom website based on a modern framework or CMS, it's only a matter of time before you run out of memory. You need to cut that MaxClients in half, if not more, in order to keep your server stable.

@hybinet:

You need to cut that MaxClients in half, if not more, in order to keep your server stable.

Good idea. I have about 5 wordpress websites on there with lots of PHP and Ajax.

I've just set maxclients to 15 - let's see if it will still run out of memory this way.

Thanks!