Apache allocation errors
I've been trying some load testing on a website, however I seem to be getting in trouble earlier than expected. I'm on a Linode 512, and once memory usage goes over 310-320M (out of 494) Apache becomes unavailable and crashes. This shows up in the log:
May 11 23:22:41 nuvini kernel: Pid: 10646, comm: apache2 Not tainted 3.0.18-linode43 #1
May 11 23:22:41 nuvini kernel: Call Trace:
May 11 23:22:41 nuvini kernel: [<c01962b8>] ? warn_alloc_failed+0x98/0x100
May 11 23:22:41 nuvini kernel: [<c0196b40>] ? __alloc_pages_nodemask+0x410/0x680
May 11 23:22:41 nuvini kernel: [<c06d5e21>] ? _raw_spin_unlock_irqrestore+0x11/0x20
May 11 23:22:41 nuvini kernel: [<c0599702>] ? __nf_ct_refresh_acct+0xc2/0xd0
May 11 23:22:41 nuvini kernel: [<c01be243>] ? T.837+0xb3/0x2e0
May 11 23:22:41 nuvini kernel: [<c01be6b6>] ? cache_alloc_refill+0x246/0x290
May 11 23:22:41 nuvini kernel: [<c01be7bd>] ? __kmalloc+0xbd/0xd0
May 11 23:22:41 nuvini kernel: [<c055027e>] ? pskb_expand_head+0x12e/0x210
May 11 23:22:41 nuvini kernel: [<c01bd921>] ? kmem_cache_free+0x41/0x60
May 11 23:22:41 nuvini kernel: [<c067b32d>] ? packet_rcv_spkt+0xfd/0x140
May 11 23:22:41 nuvini kernel: [<c05507dd>] ? __pskb_pull_tail+0x4d/0x2b0
May 11 23:22:41 nuvini kernel: [<c061d9c3>] ? ipv4_confirm+0xf3/0x1b0
May 11 23:22:41 nuvini kernel: [<c05591bd>] ? dev_hard_start_xmit+0x1dd/0x3e0
May 11 23:22:41 nuvini kernel: [<c05dde70>] ? ip_finish_output2+0x260/0x260
May 11 23:22:41 nuvini kernel: [<c05dde70>] ? ip_finish_output2+0x260/0x260
May 11 23:22:41 nuvini kernel: [<c056d582>] ? sch_direct_xmit+0xb2/0x170
May 11 23:22:41 nuvini kernel: [<c05594b9>] ? dev_queue_xmit+0xf9/0x320
May 11 23:22:41 nuvini kernel: [<c05ddfab>] ? ip_finish_output+0x13b/0x300
May 11 23:22:41 nuvini kernel: [<c05de21a>] ? ip_output+0xaa/0xe0
May 11 23:22:41 nuvini kernel: [<c05dd3b8>] ? ip_local_out+0x18/0x20
May 11 23:22:41 nuvini kernel: [<c05f0d85>] ? tcp_transmit_skb+0x385/0x670
May 11 23:22:41 nuvini kernel: [<c05f37c5>] ? tcp_write_xmit+0x1e5/0x4f0
May 11 23:22:41 nuvini kernel: [<c0106404>] ? check_events+0x8/0xc
May 11 23:22:41 nuvini kernel: [<c05f3afd>] ? tcp_push_one+0x2d/0x40
May 11 23:22:41 nuvini kernel: [<c05e6b20>] ? tcp_sendmsg+0xa40/0xb50
May 11 23:22:41 nuvini kernel: [<c0603c5c>] ? inet_recvmsg+0x5c/0xb0
May 11 23:22:41 nuvini kernel: [<c0604bc7>] ? inet_sendmsg+0x47/0xb0
May 11 23:22:41 nuvini kernel: [<c0547cfe>] ? sock_aio_write+0x11e/0x170
May 11 23:22:41 nuvini kernel: [<c0547be0>] ? sock_aio_dtor+0x10/0x10
May 11 23:22:41 nuvini kernel: [<c01c6cd6>] ? do_sync_readv_writev+0xb6/0xf0
May 11 23:22:41 nuvini kernel: [<c01c6f56>] ? rw_verify_area+0x66/0x120
May 11 23:22:41 nuvini kernel: [<c01c745a>] ? do_readv_writev+0xaa/0x1a0
May 11 23:22:41 nuvini kernel: [<c0547be0>] ? sock_aio_dtor+0x10/0x10
May 11 23:22:41 nuvini kernel: [<c0159b03>] ? getnstimeofday+0x43/0x100
May 11 23:22:41 nuvini kernel: [<c01c758e>] ? vfs_writev+0x3e/0x60
May 11 23:22:41 nuvini kernel: [<c01c76b1>] ? sys_writev+0x41/0xa0
May 11 23:22:41 nuvini kernel: [<c06d6101>] ? syscall_call+0x7/0xb
May 11 23:22:41 nuvini kernel: [<c06d0000>] ? init_intel+0xa6/0x294
May 11 23:22:41 nuvini kernel: Mem-Info:
May 11 23:22:41 nuvini kernel: DMA per-cpu:
May 11 23:22:41 nuvini kernel: CPU 0: hi: 0, btch: 1 usd: 0
May 11 23:22:41 nuvini kernel: CPU 1: hi: 0, btch: 1 usd: 0
May 11 23:22:41 nuvini kernel: CPU 2: hi: 0, btch: 1 usd: 0
May 11 23:22:41 nuvini kernel: CPU 3: hi: 0, btch: 1 usd: 0
May 11 23:22:41 nuvini kernel: Normal per-cpu:
May 11 23:22:41 nuvini kernel: CPU 0: hi: 186, btch: 31 usd: 142
May 11 23:22:41 nuvini kernel: CPU 1: hi: 186, btch: 31 usd: 169
May 11 23:22:41 nuvini kernel: CPU 2: hi: 186, btch: 31 usd: 163
May 11 23:22:41 nuvini kernel: CPU 3: hi: 186, btch: 31 usd: 13
May 11 23:22:41 nuvini kernel: active_anon:45568 inactive_anon:21809 isolated_anon:0
May 11 23:22:41 nuvini kernel: active_file:32391 inactive_file:7002 isolated_file:0
May 11 23:22:41 nuvini kernel: unevictable:0 dirty:62 writeback:0 unstable:0
May 11 23:22:41 nuvini kernel: free:7479 slab_reclaimable:5599 slab_unreclaimable:2474
May 11 23:22:41 nuvini kernel: mapped:4846 shmem:292 pagetables:696 bounce:0
May 11 23:22:41 nuvini kernel: DMA free:3976kB min:152kB low:188kB high:228kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:2988kB slab_unreclaimable:208kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3 all_unreclaimable? no
May 11 23:22:41 nuvini kernel: lowmem_reserve[]: 0 500 500 500
May 11 23:22:41 nuvini kernel: Normal free:25940kB min:4964kB low:6204kB high:7444kB active_anon:182272kB inactive_anon:87236kB active_file:129564kB inactive_file:27996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:512064kB mlocked:0kB dirty:248kB writeback:0kB mapped:19384kB shmem:1168kB slab_reclaimable:19408kB slab_unreclaimable:9688kB kernel_stack:1472kB pagetables:2784kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:76 all_unreclaimable? no
May 11 23:22:41 nuvini kernel: lowmem_reserve[]: 0 0 0 0
May 11 23:22:41 nuvini kernel: DMA: 426*4kB 146*8kB 49*16kB 10*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3976kB
May 11 23:22:41 nuvini kernel: Normal: 2103*4kB 1645*8kB 237*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 25940kB
May 11 23:22:41 nuvini kernel: 41662 total pagecache pages
May 11 23:22:41 nuvini kernel: 1993 pages in swap cache
May 11 23:22:41 nuvini kernel: Swap cache stats: add 382451, delete 380458, find 150459646/150494677
May 11 23:22:41 nuvini kernel: Free swap = 491148kB
May 11 23:22:41 nuvini kernel: Total swap = 524284kB
May 11 23:22:41 nuvini kernel: 133104 pages RAM
May 11 23:22:41 nuvini kernel: 0 pages HighMem
May 11 23:22:41 nuvini kernel: 6385 pages reserved
May 11 23:22:41 nuvini kernel: 47388 pages shared
May 11 23:22:41 nuvini kernel: 104297 pages non-shared</c06d0000></c06d6101></c01c76b1></c01c758e></c0159b03></c0547be0></c01c745a></c01c6f56></c01c6cd6></c0547be0></c0547cfe></c0604bc7></c0603c5c></c05e6b20></c05f3afd></c0106404></c05f37c5></c05f0d85></c05dd3b8></c05de21a></c05ddfab></c05594b9></c056d582></c05dde70></c05dde70></c05591bd></c061d9c3></c05507dd></c067b32d></c01bd921></c055027e></c01be7bd></c01be6b6></c01be243></c0599702></c06d5e21></c0196b40></c01962b8>
My settings in apache:
<ifmodule mpm_prefork_module="">StartServers 3
MinSpareServers 3
MaxSpareServers 6
MaxClients 15
ServerLimit 15
MaxRequestsPerChild 2000</ifmodule>
I googled around and found this fix:
Which said to do````
vm.minfreekbytes = 5120.
````in the sysctl file.
I did this, followed by sysctl -p, however the problem stays the same.
Anything I'm doing wrong? Thanks!
5 Replies
Is there a reason I can't go beyond that? Shouldn't I be able to raise MaxClients and utilize at least another 130M (and leave about 50 just for some breathing room)
I use htop to show me memory usage, if it matters.
@Obsidian:
Out of curiosity, are you running a database server on that too? There may be other things needing the leftover memory (along with the kernel itself).
Yes, it's also running MySQL.
I would however expect htop to show me that all the memory is being used. ATM is just shows me ~300M used under load. I would understand it if it told me the RAM was being filled completely, but it simply doesn't look that way.
A new problem arises though, I can now kill my site (or at least, DDoS) it using apachebench and even holding in F5 on the site will raise all CPUs to 100% usage. (but memory stays low)
Is this also caused by the workermpm settings in apache.conf? I used a calculation tool for prefork, but I can't find a safe/optimal value for workermpm. Any ideas on what to use?
(Or is this completely unrelated to those settings?)
Edit: I should note that it is not Apache going up, it's all the php-fpm processes.