View All Products
Compute
Storage
Databases
Networking
Developer Tools
Delivery
Security
Services
Industries
Pricing
Community
Engage With Us
Community ▶ Questions ▶ Out of memory error...but I'm not out of memory.
Log in to Ask a Question

Out of memory error...but I'm not out of memory.

general

Every Saturday morning just after midnight (eastern), I start getting out of memory errors but the system has plenty of free memory. Rebooting the Linode solves the problem until the next week. Here are some of the errors including some kernel errors which may be the source:

Jun 22 09:01:45 server mysqld: 130622 9:01:45 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:01:49 server mysqld: 130622 9:01:49 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:01:52 server mysqld: 130622 9:01:52 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:02:49 server mysqld: 130622 9:02:49 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:02:49 server mysqld: 130622 9:02:49 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:05:36 server mysqld: 130622 9:05:36 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:06:09 server mysqld: 130622 9:06:09 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)

Jun 22 09:07:25 server kernel: released 0 pages of unused memory

Jun 22 09:07:25 server kernel: initial memory mapped : 0 - 00b52000

Jun 22 09:07:25 server kernel: Base memory trampoline at [c009f000] 9f000 size 4096

Jun 22 09:07:25 server kernel: initmemorymapping: 0000000000000000-000000002d3fe000

Jun 22 09:07:25 server kernel: virtual kernel memory layout:

Jun 22 09:07:25 server kernel: Freeing unused kernel memory: 412k freed

Jun 22 09:07:29 server mysqld: 130622 9:07:29 InnoDB: The InnoDB memory heap is disabled

The amount of memory 'needed' is always the same number of bytes which is a clear indication that there isn't an actual shortage of memory. Has anyone ever seen this or have any ideas what may be causing it?

13 Replies

@Poieo:

The amount of memory 'needed' is always the same number of bytes which is a clear indication that there isn't an actual shortage of memory.

This is incorrect. The amount of memory "needed" is always the same and that number has nothing to do with what memory you have available or not.

Maybe a larger query that runs every so often? In this case, MySQL is looking for 8.3 Megs of memory. Define "plenty of free".

> Every Saturday morning just after midnight

Look at syslog just before the crash / river of nulls, and post any cron jobs mentioned soon before the OOM.

Alternatively,

ls /etc/cron.weekly

cat /etc/crontab

cat /etc/cron.d/*

cat /var/spool/cron/crontabs/*

Let's start with most obvious suspect, you know. :)

Oops.

Gods, remove your passwords from that post, and change them on the server!

Worseover, you don't seem to have anything scheduled explicitly on Saturday - there's cron.weekly on Sunday and barracuda on Monday, but everything else is scheduled much more often - at a specific hour, or multiple times per hour.

Including some stuff running every minute, that'll definitely be a lot of noise in the logs.

Sadly, this may mean that it's one of these "frequent" jobs that causes the OOM, but at this specific time only due to some internal logic making it execute some different operation, or work on different data when ran on Saturday.

What I can say - if you don't have a system monitoring system like munin installed and I guess you don't, you can do something like this on Friday afternoon:

Open a new SSH session, run

nohup sh -c 'while true; do date; free; sleep 30; done >> ~/mem_stats' >/dev/null &

then just close the terminal (or run "exit" twice in a row, to let you exit despite having a background job running).

It'll append the timestamp and memory statistics into the mem_stats file in your homedir every half a minute, so when it OOMs on Saturday again, you should be able to open this file, and look up at which specific minute the memory usage started to skyrocket.

Then you can correlate that timestamp with your logs;

  • syslog to see cron jobs that were started

  • webserver access log to see what kind of requests your website received at the time

  • your application's own logs if stuff like "daily.sh" writes them, to see what does it do at the time

etc.

That should allow you to narrow it to a small number of suspects. Then try disabling them before next Saturday, and if it helped, enable back one by one untill you'll find it.

Or you could try running them manually (from the right account, mind!) to see what happens.

Good luck.

And - don't take my word as gospel - points at post signature - someone else might have a better idea. :)

Well, I officially forgot to do this. The errors started again on Saturday around 9am this week.

Here is a link to a screen shot of my memory usage when the errors were occurring. As you can see I wasn't out of memory, but maybe I'm looking at the wrong thing as this is server RAM and the errors are php memory. I looked over all my server stats when the errors were happening and couldn't see anything wrong, but it may be more narrow than the stats would show.

https://dl.dropboxusercontent.com/u/203 … 3%20PM.png">https://dl.dropboxusercontent.com/u/20321169/Screen%20Shot%202013-06-29%20at%204.45.53%20PM.png

I'll try to remember to do this next week.

Poieo,

What are your RAM limits for PHP? They're set in your php.ini file and/or your .htaccess file. I think the setting is called "memory_limit". I believe the entry in your site's .htaccess file overrides the setting in your server's php.ini file.

James

Considering you had errors in mysql being unable to allocate RAM, the problem is not limited to PHP.

(Or were you asking whether PHP is restricted, or allowed to allocate a ton of RAM on demand, James?)

Also, munin reads system stats every 5 minutes by default - so if it climbed to the roof within this time, you wouldn't see it on the graph.

You have, as we just seen on the screenshot, a very beefy server (8G?), and if I remember these cronjobs you posted in the past correctly, they're being executed very often and are (probably) necessary, so you can't just disable them.

However, could you post the syslog and webserver logs from last ~10-15 minutes before the OOM?

Here is 10 minutes of syslog before this week's first error. I don't think there is any sensitive info in here. Please let me know if you see any. I'll also post last weeks:

Jun 29 09:42:01 server /USR/SBIN/CRON[9143]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:42:01 server /USR/SBIN/CRON[9144]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:42:01 server /USR/SBIN/CRON[9145]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:42:01 server /USR/SBIN/CRON[9146]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:42:01 server /USR/SBIN/CRON[9147]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:42:01 server /USR/SBIN/CRON[9148]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:42:03 server pure-ftpd: (?@142.4.107.135) [INFO] New connection from 142.4.107.135
Jun 29 09:42:03 server pure-ftpd: (?@142.4.107.135) [DEBUG] Command [user] [scinsuranceagent.com]
Jun 29 09:42:03 server pure-ftpd: (?@142.4.107.135) [WARNING] Sorry, cleartext sessions are not accepted on this server.#012Please reconnect using SSL/TLS security mechanisms.
Jun 29 09:43:01 server /USR/SBIN/CRON[9871]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:43:01 server /USR/SBIN/CRON[9872]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:43:01 server /USR/SBIN/CRON[9873]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:43:01 server /USR/SBIN/CRON[9874]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:43:01 server /USR/SBIN/CRON[9875]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:43:01 server /USR/SBIN/CRON[9876]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:43:48 server kernel: swapper: page allocation failure. order:4, mode:0x20
Jun 29 09:43:48 server kernel: Pid: 0, comm: swapper Not tainted 2.6.39.1-linode34 #1
Jun 29 09:43:48 server kernel: Call Trace:
Jun 29 09:43:48 server kernel: [<c0189a30>] ? __alloc_pages_nodemask+0x530/0x6f0
Jun 29 09:43:48 server kernel: [<c01afb13>] ? T.819+0xb3/0x2e0
Jun 29 09:43:48 server kernel: [<c01aff86>] ? cache_alloc_refill+0x246/0x290
Jun 29 09:43:48 server kernel: [<c0139826>] ? local_bh_enable+0x16/0x80
Jun 29 09:43:48 server kernel: [<c01b008d>] ? __kmalloc+0xbd/0xd0
Jun 29 09:43:48 server kernel: [<c050f07e>] ? pskb_expand_head+0x12e/0x200
Jun 29 09:43:48 server kernel: [<c050f5bd>] ? __pskb_pull_tail+0x4d/0x2b0
Jun 29 09:43:48 server kernel: [<c05d9263>] ? ipv4_confirm+0xd3/0x180
Jun 29 09:43:48 server kernel: [<c0517d6d>] ? dev_hard_start_xmit+0x1dd/0x3e0
Jun 29 09:43:48 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 29 09:43:48 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 29 09:43:48 server kernel: [<c052bcc2>] ? sch_direct_xmit+0xb2/0x170
Jun 29 09:43:48 server kernel: [<c0518069>] ? dev_queue_xmit+0xf9/0x320
Jun 29 09:43:48 server kernel: [<c059aa3b>] ? ip_finish_output+0x13b/0x300
Jun 29 09:43:48 server kernel: [<c059acaa>] ? ip_output+0xaa/0xe0
Jun 29 09:43:48 server kernel: [<c0599e78>] ? ip_local_out+0x18/0x20
Jun 29 09:43:48 server kernel: [<c059a257>] ? ip_queue_xmit+0x117/0x3d0
Jun 29 09:43:48 server kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Jun 29 09:43:48 server kernel: [<c01062c4>] ? check_events+0x8/0xc
Jun 29 09:43:48 server kernel: [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Jun 29 09:43:48 server kernel: [<c01b004f>] ? __kmalloc+0x7f/0xd0
Jun 29 09:43:48 server kernel: [<c05ad70f>] ? tcp_transmit_skb+0x37f/0x660
Jun 29 09:43:48 server kernel: [<c05b0165>] ? tcp_write_xmit+0x1e5/0x4f0
Jun 29 09:43:48 server kernel: [<c05b04d4>] ? __tcp_push_pending_frames+0x24/0x90
Jun 29 09:43:48 server kernel: [<c05ac20b>] ? tcp_rcv_established+0xfb/0x610
Jun 29 09:43:48 server kernel: [<c0552774>] ? nf_iterate+0x74/0xa0
Jun 29 09:43:48 server kernel: [<c05b2fee>] ? tcp_v4_do_rcv+0xce/0x170
Jun 29 09:43:48 server kernel: [<c05b3749>] ? tcp_v4_rcv+0x6b9/0x7a0
Jun 29 09:43:48 server kernel: [<c0595887>] ? ip_local_deliver_finish+0x97/0x220
Jun 29 09:43:48 server kernel: [<c05957f0>] ? ip_rcv+0x320/0x320
Jun 29 09:43:48 server kernel: [<c059524b>] ? ip_rcv_finish+0xfb/0x380
Jun 29 09:43:48 server kernel: [<c0516ca9>] ? __netif_receive_skb+0x339/0x3d0
Jun 29 09:43:48 server kernel: [<c0516f47>] ? netif_receive_skb+0x67/0x70
Jun 29 09:43:48 server kernel: [<c04b84cc>] ? handle_incoming_queue+0x17c/0x250
Jun 29 09:43:48 server kernel: [<c04b87bc>] ? xennet_poll+0x21c/0x540
Jun 29 09:43:48 server kernel: [<c0131061>] ? load_balance+0x71/0x590
Jun 29 09:43:48 server kernel: [<c05176da>] ? net_rx_action+0xea/0x190
Jun 29 09:43:48 server kernel: [<c013956c>] ? __do_softirq+0x7c/0x110
Jun 29 09:43:48 server kernel: [<c01394f0>] ? __local_bh_enable+0x70/0x70
Jun 29 09:43:48 server kernel: <irq>[<c013944e>] ? irq_exit+0x6e/0x90
Jun 29 09:43:48 server kernel: [<c044d14d>] ? xen_evtchn_do_upcall+0x1d/0x30
Jun 29 09:43:48 server kernel: [<c0690c07>] ? xen_do_upcall+0x7/0xc
Jun 29 09:43:48 server kernel: [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Jun 29 09:43:48 server kernel: [<c0105b3f>] ? xen_safe_halt+0xf/0x20
Jun 29 09:43:48 server kernel: [<c010f1ff>] ? default_idle+0x2f/0x60
Jun 29 09:43:48 server kernel: [<c0107e52>] ? cpu_idle+0x42/0x70
Jun 29 09:43:48 server kernel: [<c0830797>] ? start_kernel+0x2c8/0x2cd
Jun 29 09:43:48 server kernel: [<c083030d>] ? kernel_init+0x126/0x126
Jun 29 09:43:48 server kernel: [<c083395b>] ? xen_start_kernel+0x4f7/0x4ff
Jun 29 09:43:48 server kernel: Mem-Info:
Jun 29 09:43:48 server kernel: DMA per-cpu:
Jun 29 09:43:48 server kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    2: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    3: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    4: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    5: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    6: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: CPU    7: hi:    0, btch:   1 usd:   0
Jun 29 09:43:48 server kernel: Normal per-cpu:
Jun 29 09:43:48 server kernel: CPU    0: hi:  186, btch:  31 usd: 155
Jun 29 09:43:48 server kernel: CPU    1: hi:  186, btch:  31 usd: 162
Jun 29 09:43:48 server kernel: CPU    2: hi:  186, btch:  31 usd: 162
Jun 29 09:43:48 server kernel: CPU    3: hi:  186, btch:  31 usd:  83
Jun 29 09:43:48 server kernel: CPU    4: hi:  186, btch:  31 usd: 155
Jun 29 09:43:48 server kernel: CPU    5: hi:  186, btch:  31 usd: 172
Jun 29 09:43:48 server kernel: CPU    6: hi:  186, btch:  31 usd: 165
Jun 29 09:43:48 server kernel: CPU    7: hi:  186, btch:  31 usd: 184
Jun 29 09:43:48 server kernel: HighMem per-cpu:
Jun 29 09:43:48 server kernel: CPU    0: hi:  186, btch:  31 usd:  16
Jun 29 09:43:48 server kernel: CPU    1: hi:  186, btch:  31 usd: 177
Jun 29 09:43:48 server kernel: CPU    2: hi:  186, btch:  31 usd: 174
Jun 29 09:43:48 server kernel: CPU    3: hi:  186, btch:  31 usd:  48
Jun 29 09:43:48 server kernel: CPU    4: hi:  186, btch:  31 usd: 163
Jun 29 09:43:48 server kernel: CPU    5: hi:  186, btch:  31 usd: 163
Jun 29 09:43:48 server kernel: CPU    6: hi:  186, btch:  31 usd: 181
Jun 29 09:43:48 server kernel: CPU    7: hi:  186, btch:  31 usd: 160
Jun 29 09:43:48 server kernel: active_anon:767188 inactive_anon:114278 isolated_anon:0
Jun 29 09:43:48 server kernel: active_file:251230 inactive_file:235303 isolated_file:0
Jun 29 09:43:48 server kernel: unevictable:0 dirty:490 writeback:0 unstable:0
Jun 29 09:43:48 server kernel: free:650501 slab_reclaimable:35681 slab_unreclaimable:6250
Jun 29 09:43:48 server kernel: mapped:112768 shmem:158756 pagetables:8714 bounce:0
Jun 29 09:43:48 server kernel: DMA free:2052kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 29 09:43:48 server kernel: lowmem_reserve[]: 0 702 8112 8112
Jun 29 09:43:48 server kernel: Normal free:5292kB min:3352kB low:4188kB high:5028kB active_anon:0kB inactive_anon:0kB active_file:207708kB inactive_file:227044kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:719320kB mlocked:0kB dirty:460kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:142724kB slab_unreclaimable:25000kB kernel_stack:1712kB pagetables:34856kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 29 09:43:48 server kernel: lowmem_reserve[]: 0 0 59277 59277
Jun 29 09:43:48 server kernel: HighMem free:2594660kB min:512kB low:9356kB high:18200kB active_anon:3068752kB inactive_anon:457112kB active_file:797212kB inactive_file:714168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7587492kB mlocked:0kB dirty:1500kB writeback:0kB mapped:451072kB shmem:635024kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 29 09:43:48 server kernel: lowmem_reserve[]: 0 0 0 0
Jun 29 09:43:48 server kernel: DMA: 5*4kB 4*8kB 5*16kB 2*32kB 5*64kB 2*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2052kB
Jun 29 09:43:48 server kernel: Normal: 779*4kB 6*8kB 96*16kB 15*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5244kB
Jun 29 09:43:48 server kernel: HighMem: 4744*4kB 34225*8kB 54864*16kB 18104*32kB 5615*64kB 1923*128kB 706*256kB 80*512kB 13*1024kB 0*2048kB 1*4096kB = 2594536kB
Jun 29 09:43:48 server kernel: 645288 total pagecache pages
Jun 29 09:43:48 server kernel: 0 pages in swap cache
Jun 29 09:43:48 server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 29 09:43:48 server kernel: Free swap  = 1048572kB
Jun 29 09:43:48 server kernel: Total swap = 1048572kB
Jun 29 09:43:48 server kernel: 2097136 pages RAM
Jun 29 09:43:48 server kernel: 1911810 pages HighMem
Jun 29 09:43:48 server kernel: 21134 pages reserved
Jun 29 09:43:48 server kernel: 1805403 pages shared
Jun 29 09:43:48 server kernel: 1119134 pages non-shared
Jun 29 09:44:01 server /USR/SBIN/CRON[10531]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:44:01 server /USR/SBIN/CRON[10532]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:44:01 server /USR/SBIN/CRON[10533]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:44:01 server /USR/SBIN/CRON[10534]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:44:01 server /USR/SBIN/CRON[10535]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:44:01 server /USR/SBIN/CRON[10537]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:44:09 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.239 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=10327 DF PROTO=TCP SPT=3716 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:44:09 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.250 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=10338 DF PROTO=TCP SPT=3727 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:44:12 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.239 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=13391 DF PROTO=TCP SPT=3716 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:44:12 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.250 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=13508 DF PROTO=TCP SPT=3727 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:44:18 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.239 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=16333 DF PROTO=TCP SPT=3716 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:44:18 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=146.0.74.29 DST=74.207.229.250 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=16449 DF PROTO=TCP SPT=3727 DPT=8118 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 29 09:45:01 server /USR/SBIN/CRON[11184]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:45:01 server /USR/SBIN/CRON[11186]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 29 09:45:01 server /USR/SBIN/CRON[11185]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:45:01 server /USR/SBIN/CRON[11187]: (root) CMD (bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1)
Jun 29 09:45:01 server /USR/SBIN/CRON[11188]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:45:01 server /USR/SBIN/CRON[11189]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:45:01 server /USR/SBIN/CRON[11192]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:45:01 server /USR/SBIN/CRON[11193]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:46:01 server /USR/SBIN/CRON[12594]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:46:01 server /USR/SBIN/CRON[12595]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:46:01 server /USR/SBIN/CRON[12596]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:46:01 server /USR/SBIN/CRON[12598]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:46:01 server /USR/SBIN/CRON[12600]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:46:01 server /USR/SBIN/CRON[12599]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:46:30 server drupal: http://aegir.poieohosting.com|1372513590|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:31 server drupal: http://aegir.poieohosting.com|1372513591|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:32 server drupal: http://aegir.poieohosting.com|1372513592|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:33 server drupal: http://aegir.poieohosting.com|1372513593|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:33 server drupal: http://aegir.poieohosting.com|1372513593|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:34 server drupal: http://aegir.poieohosting.com|1372513594|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:35 server drupal: http://aegir.poieohosting.com|1372513595|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:36 server drupal: http://aegir.poieohosting.com|1372513596|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:37 server drupal: http://aegir.poieohosting.com|1372513597|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:37 server drupal: http://aegir.poieohosting.com|1372513597|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:38 server drupal: http://aegir.poieohosting.com|1372513598|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:46:39 server drupal: http://aegir.poieohosting.com|1372513599|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 29 09:47:00 server kernel: swapper: page allocation failure. order:4, mode:0x20
Jun 29 09:47:00 server kernel: Pid: 0, comm: swapper Not tainted 2.6.39.1-linode34 #1
Jun 29 09:47:00 server kernel: Call Trace:
Jun 29 09:47:00 server kernel: [<c0189a30>] ? __alloc_pages_nodemask+0x530/0x6f0
Jun 29 09:47:00 server kernel: [<c01afb13>] ? T.819+0xb3/0x2e0
Jun 29 09:47:00 server kernel: [<c01aff86>] ? cache_alloc_refill+0x246/0x290
Jun 29 09:47:00 server kernel: [<c0139826>] ? local_bh_enable+0x16/0x80
Jun 29 09:47:00 server kernel: [<c01b008d>] ? __kmalloc+0xbd/0xd0
Jun 29 09:47:00 server kernel: [<c050f07e>] ? pskb_expand_head+0x12e/0x200
Jun 29 09:47:00 server kernel: [<c050f5bd>] ? __pskb_pull_tail+0x4d/0x2b0
Jun 29 09:47:00 server kernel: [<c05d9263>] ? ipv4_confirm+0xd3/0x180
Jun 29 09:47:00 server kernel: [<c0517d6d>] ? dev_hard_start_xmit+0x1dd/0x3e0
Jun 29 09:47:00 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 29 09:47:00 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 29 09:47:00 server kernel: [<c052bcc2>] ? sch_direct_xmit+0xb2/0x170
Jun 29 09:47:00 server kernel: [<c0518069>] ? dev_queue_xmit+0xf9/0x320
Jun 29 09:47:00 server kernel: [<c059aa3b>] ? ip_finish_output+0x13b/0x300
Jun 29 09:47:00 server kernel: [<c059acaa>] ? ip_output+0xaa/0xe0
Jun 29 09:47:00 server kernel: [<c0599e78>] ? ip_local_out+0x18/0x20
Jun 29 09:47:00 server kernel: [<c059a257>] ? ip_queue_xmit+0x117/0x3d0
Jun 29 09:47:00 server kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Jun 29 09:47:00 server kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Jun 29 09:47:00 server kernel: [<c01062c4>] ? check_events+0x8/0xc
Jun 29 09:47:00 server kernel: [<c05ad70f>] ? tcp_transmit_skb+0x37f/0x660
Jun 29 09:47:00 server kernel: [<c05b0165>] ? tcp_write_xmit+0x1e5/0x4f0
Jun 29 09:47:00 server kernel: [<c05b04d4>] ? __tcp_push_pending_frames+0x24/0x90
Jun 29 09:47:00 server kernel: [<c05ac4e2>] ? tcp_rcv_established+0x3d2/0x610
Jun 29 09:47:00 server kernel: [<c05b2fee>] ? tcp_v4_do_rcv+0xce/0x170
Jun 29 09:47:00 server kernel: [<c05b3749>] ? tcp_v4_rcv+0x6b9/0x7a0
Jun 29 09:47:00 server kernel: [<c0595887>] ? ip_local_deliver_finish+0x97/0x220
Jun 29 09:47:00 server kernel: [<c05957f0>] ? ip_rcv+0x320/0x320
Jun 29 09:47:00 server kernel: [<c059524b>] ? ip_rcv_finish+0xfb/0x380
Jun 29 09:47:00 server kernel: [<c0516ca9>] ? __netif_receive_skb+0x339/0x3d0
Jun 29 09:47:00 server kernel: [<c0516f47>] ? netif_receive_skb+0x67/0x70
Jun 29 09:47:00 server kernel: [<c04b84cc>] ? handle_incoming_queue+0x17c/0x250
Jun 29 09:47:00 server kernel: [<c04b87bc>] ? xennet_poll+0x21c/0x540
Jun 29 09:47:00 server kernel: [<c0131061>] ? load_balance+0x71/0x590
Jun 29 09:47:00 server kernel: [<c05176da>] ? net_rx_action+0xea/0x190
Jun 29 09:47:00 server kernel: [<c013956c>] ? __do_softirq+0x7c/0x110
Jun 29 09:47:00 server kernel: [<c01394f0>] ? __local_bh_enable+0x70/0x70
Jun 29 09:47:00 server kernel: <irq>[<c013944e>] ? irq_exit+0x6e/0x90
Jun 29 09:47:00 server kernel: [<c044d14d>] ? xen_evtchn_do_upcall+0x1d/0x30
Jun 29 09:47:00 server kernel: [<c0690c07>] ? xen_do_upcall+0x7/0xc
Jun 29 09:47:00 server kernel: [<c01013a7>] ? hypercall_page+0x3a7/0x1000
Jun 29 09:47:00 server kernel: [<c0105b3f>] ? xen_safe_halt+0xf/0x20
Jun 29 09:47:00 server kernel: [<c010f1ff>] ? default_idle+0x2f/0x60
Jun 29 09:47:00 server kernel: [<c0107e52>] ? cpu_idle+0x42/0x70
Jun 29 09:47:00 server kernel: [<c0830797>] ? start_kernel+0x2c8/0x2cd
Jun 29 09:47:00 server kernel: [<c083030d>] ? kernel_init+0x126/0x126
Jun 29 09:47:00 server kernel: [<c083395b>] ? xen_start_kernel+0x4f7/0x4ff
Jun 29 09:47:00 server kernel: Mem-Info:
Jun 29 09:47:00 server kernel: DMA per-cpu:
Jun 29 09:47:00 server kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    2: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    3: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    4: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    5: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    6: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: CPU    7: hi:    0, btch:   1 usd:   0
Jun 29 09:47:00 server kernel: Normal per-cpu:
Jun 29 09:47:00 server kernel: CPU    0: hi:  186, btch:  31 usd: 154
Jun 29 09:47:00 server kernel: CPU    1: hi:  186, btch:  31 usd: 159
Jun 29 09:47:00 server kernel: CPU    2: hi:  186, btch:  31 usd: 165
Jun 29 09:47:00 server kernel: CPU    3: hi:  186, btch:  31 usd: 143
Jun 29 09:47:00 server kernel: CPU    4: hi:  186, btch:  31 usd: 109
Jun 29 09:47:00 server kernel: CPU    5: hi:  186, btch:  31 usd: 151
Jun 29 09:47:00 server kernel: CPU    6: hi:  186, btch:  31 usd:  60
Jun 29 09:47:00 server kernel: CPU    7: hi:  186, btch:  31 usd:  59
Jun 29 09:47:00 server kernel: HighMem per-cpu:
Jun 29 09:47:00 server kernel: CPU    0: hi:  186, btch:  31 usd: 179
Jun 29 09:47:00 server kernel: CPU    1: hi:  186, btch:  31 usd: 160
Jun 29 09:47:00 server kernel: CPU    2: hi:  186, btch:  31 usd: 172
Jun 29 09:47:00 server kernel: CPU    3: hi:  186, btch:  31 usd: 160
Jun 29 09:47:00 server kernel: CPU    4: hi:  186, btch:  31 usd: 158
Jun 29 09:47:00 server kernel: CPU    5: hi:  186, btch:  31 usd: 183
Jun 29 09:47:00 server kernel: CPU    6: hi:  186, btch:  31 usd:  50
Jun 29 09:47:00 server kernel: CPU    7: hi:  186, btch:  31 usd: 118
Jun 29 09:47:00 server kernel: active_anon:768434 inactive_anon:114278 isolated_anon:0
Jun 29 09:47:00 server kernel: active_file:251230 inactive_file:231836 isolated_file:0
Jun 29 09:47:00 server kernel: unevictable:0 dirty:937 writeback:0 unstable:0
Jun 29 09:47:00 server kernel: free:652880 slab_reclaimable:35680 slab_unreclaimable:6305
Jun 29 09:47:00 server kernel: mapped:112769 shmem:158756 pagetables:8672 bounce:0
Jun 29 09:47:00 server kernel: DMA free:2052kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 29 09:47:00 server kernel: lowmem_reserve[]: 0 702 8112 8112
Jun 29 09:47:00 server kernel: Normal free:8360kB min:3352kB low:4188kB high:5028kB active_anon:0kB inactive_anon:0kB active_file:207776kB inactive_file:225052kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:719320kB mlocked:0kB dirty:320kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:142720kB slab_unreclaimable:25220kB kernel_stack:1648kB pagetables:34688kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 29 09:47:00 server kernel: lowmem_reserve[]: 0 0 59277 59277
Jun 29 09:47:00 server kernel: HighMem free:2601108kB min:512kB low:9356kB high:18200kB active_anon:3073736kB inactive_anon:457112kB active_file:797144kB inactive_file:702292kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7587492kB mlocked:0kB dirty:3428kB writeback:0kB mapped:451076kB shmem:635024kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 29 09:47:00 server kernel: lowmem_reserve[]: 0 0 0 0
Jun 29 09:47:00 server kernel: DMA: 5*4kB 4*8kB 5*16kB 2*32kB 5*64kB 2*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2052kB
Jun 29 09:47:00 server kernel: Normal: 896*4kB 190*8kB 161*16kB 18*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8320kB
Jun 29 09:47:00 server kernel: HighMem: 9815*4kB 32497*8kB 54865*16kB 18103*32kB 5615*64kB 1922*128kB 707*256kB 80*512kB 13*1024kB 0*2048kB 1*4096kB = 2601108kB
Jun 29 09:47:00 server kernel: 641822 total pagecache pages
Jun 29 09:47:00 server kernel: 0 pages in swap cache
Jun 29 09:47:00 server kernel: Swap cache stats: add 0, delete 0, find 0/0
Jun 29 09:47:00 server kernel: Free swap  = 1048572kB
Jun 29 09:47:00 server kernel: Total swap = 1048572kB
Jun 29 09:47:00 server kernel: 2097136 pages RAM
Jun 29 09:47:00 server kernel: 1911810 pages HighMem
Jun 29 09:47:00 server kernel: 21134 pages reserved
Jun 29 09:47:00 server kernel: 1805175 pages shared
Jun 29 09:47:00 server kernel: 1117617 pages non-shared
Jun 29 09:47:01 server /USR/SBIN/CRON[13717]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:47:01 server /USR/SBIN/CRON[13718]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:47:01 server /USR/SBIN/CRON[13719]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:47:01 server /USR/SBIN/CRON[13720]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:47:01 server /USR/SBIN/CRON[13721]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:47:01 server /USR/SBIN/CRON[13725]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:48:01 server /USR/SBIN/CRON[14412]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:48:01 server /USR/SBIN/CRON[14413]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:48:01 server /USR/SBIN/CRON[14414]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:48:01 server /USR/SBIN/CRON[14415]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:48:01 server /USR/SBIN/CRON[14416]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:48:01 server /USR/SBIN/CRON[14419]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:48:17 server postfix/qmgr[4016]: 432B41981AE: from=<www-data@server.poieohosting.com>, size=4124, nrcpt=2 (queue active)
Jun 29 09:48:17 server postfix/qmgr[4016]: 40C7E1981AF: from=<www-data@server.poieohosting.com>, size=2574, nrcpt=2 (queue active)
Jun 29 09:48:17 server postfix/qmgr[4016]: 30F7EE5403E: from=<www-data@server.poieohosting.com>, size=5224, nrcpt=2 (queue active)
Jun 29 09:48:47 server postfix/smtp[14783]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 29 09:48:47 server postfix/smtp[14783]: 432B41981AE: to=<support@example.com>, relay=none, delay=294503, delays=294473/0.01/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 29 09:48:47 server postfix/smtp[14784]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 29 09:48:47 server postfix/smtp[14784]: 40C7E1981AF: to=<support@example.com>, relay=none, delay=294522, delays=294492/0.01/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 29 09:48:47 server postfix/smtp[14787]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 29 09:48:47 server postfix/smtp[14787]: 30F7EE5403E: to=<support@example.com>, relay=none, delay=143301, delays=143271/0.02/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 29 09:49:01 server /USR/SBIN/CRON[15062]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:49:01 server /USR/SBIN/CRON[15063]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:49:01 server /USR/SBIN/CRON[15064]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:49:01 server /USR/SBIN/CRON[15065]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:49:01 server /USR/SBIN/CRON[15066]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:49:01 server /USR/SBIN/CRON[15070]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:50:01 server /USR/SBIN/CRON[15706]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:50:01 server /USR/SBIN/CRON[15707]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:50:01 server /USR/SBIN/CRON[15708]: (root) CMD (bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1)
Jun 29 09:50:01 server /USR/SBIN/CRON[15709]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:50:01 server /USR/SBIN/CRON[15711]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:50:01 server /USR/SBIN/CRON[15710]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:50:01 server /USR/SBIN/CRON[15712]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:51:01 server /USR/SBIN/CRON[17021]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:51:01 server /USR/SBIN/CRON[17022]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:51:01 server /USR/SBIN/CRON[17023]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:51:01 server /USR/SBIN/CRON[17024]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:51:01 server /USR/SBIN/CRON[17025]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:51:01 server /USR/SBIN/CRON[17026]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:52:01 server /USR/SBIN/CRON[17735]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 29 09:52:01 server /USR/SBIN/CRON[17736]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 29 09:52:01 server /USR/SBIN/CRON[17737]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 29 09:52:01 server /USR/SBIN/CRON[17739]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 29 09:52:01 server /USR/SBIN/CRON[17738]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 29 09:52:01 server /USR/SBIN/CRON[17740]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 29 09:52:46 server mysqld: 130629  9:52:46 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)
Jun 29 09:52:47 server mysqld: 130629  9:52:47 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)</support@example.com></support@example.com></support@example.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></c083395b></c083030d></c0830797></c0107e52></c010f1ff></c0105b3f></c01013a7></c0690c07></c044d14d></c013944e></irq></c01394f0></c013956c></c05176da></c0131061></c04b87bc></c04b84cc></c0516f47></c0516ca9></c059524b></c05957f0></c0595887></c05b3749></c05b2fee></c05ac4e2></c05b04d4></c05b0165></c05ad70f></c01062c4></c0105ae7></c0105ae7></c059a257></c0599e78></c059acaa></c059aa3b></c0518069></c052bcc2></c059a900></c059a900></c0517d6d></c05d9263></c050f5bd></c050f07e></c01b008d></c0139826></c01aff86></c01afb13></c0189a30></c083395b></c083030d></c0830797></c0107e52></c010f1ff></c0105b3f></c01013a7></c0690c07></c044d14d></c013944e></irq></c01394f0></c013956c></c05176da></c0131061></c04b87bc></c04b84cc></c0516f47></c0516ca9></c059524b></c05957f0></c0595887></c05b3749></c05b2fee></c0552774></c05ac20b></c05b04d4></c05b0165></c05ad70f></c01b004f></c01062bb></c01062c4></c0105ae7></c059a257></c0599e78></c059acaa></c059aa3b></c0518069></c052bcc2></c059a900></c059a900></c0517d6d></c05d9263></c050f5bd></c050f07e></c01b008d></c0139826></c01aff86></c01afb13></c0189a30>

Here is last weeks:

Jun 22 00:00:01 server /USR/SBIN/CRON[13802]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:00:01 server /USR/SBIN/CRON[13782]: (CRON) error (grandchild #13794 failed with exit status 1)
Jun 22 00:00:04 server postfix/pickup[21855]: 4265C19804D: uid=33 from= <www-data>Jun 22 00:00:04 server postfix/cleanup[13994]: 4265C19804D: message-id=<1d14a6dc74b1e1fa1205e0cb5775b9d9@portal.poieodesign.com>
Jun 22 00:00:04 server postfix/qmgr[3872]: 4265C19804D: from=<www-data@server.poieohosting.com>, size=2520, nrcpt=2 (queue active)
Jun 22 00:00:04 server postfix/error[13996]: 4265C19804D: to=<support@example.com>, relay=none, delay=0.03, delays=0.01/0.01/0/0.01, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:00:04 server postfix/pickup[21855]: 4D94E1981B9: uid=1000 from= <drupalcron>Jun 22 00:00:04 server postfix/cleanup[13994]: 4D94E1981B9: message-id=<20130622040004.4D94E1981B9@server.poieohosting.com>
Jun 22 00:00:04 server postfix/qmgr[3872]: 4D94E1981B9: from=<drupalcron@server.poieohosting.com>, size=1320, nrcpt=1 (queue active)
Jun 22 00:00:04 server postfix/local[14005]: 4D94E1981B9: to=<drupalcron@server.poieohosting.com>, orig_to=<drupalcron>, relay=local, delay=0.04, delays=0/0.01/0/0.03, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")
Jun 22 00:00:04 server postfix/qmgr[3872]: 4D94E1981B9: removed
Jun 22 00:00:05 server postfix/smtp[13998]: 4265C19804D: to=<dave@poieodesign.com>, relay=aspmx.l.google.com[74.125.140.26]:25, delay=0.84, delays=0.01/0.01/0.07/0.75, dsn=2.0.0, status=sent (250 2.0.0 OK 1371873605 i42si4485868yhd.197 - gsmtp)
Jun 22 00:00:21 server postfix/pickup[21855]: B81861981B9: uid=33 from= <www-data>Jun 22 00:00:21 server postfix/cleanup[13994]: B81861981B9: message-id= <daf2ed4beaa389cd2504a4feebce87a6@portal.poieodesign.com>Jun 22 00:00:21 server postfix/qmgr[3872]: B81861981B9: from=<www-data@server.poieohosting.com>, size=4124, nrcpt=2 (queue active)
Jun 22 00:00:21 server postfix/error[13996]: B81861981B9: to=<support@example.com>, relay=none, delay=0.01, delays=0/0/0/0, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:00:21 server postfix/pickup[21855]: BEAFE1981C6: uid=1000 from= <drupalcron>Jun 22 00:00:21 server postfix/cleanup[13994]: BEAFE1981C6: message-id=<20130622040021.BEAFE1981C6@server.poieohosting.com>
Jun 22 00:00:21 server postfix/qmgr[3872]: BEAFE1981C6: from=<drupalcron@server.poieohosting.com>, size=2051, nrcpt=1 (queue active)
Jun 22 00:00:21 server postfix/local[14005]: BEAFE1981C6: to=<drupalcron@server.poieohosting.com>, orig_to=<drupalcron>, relay=local, delay=0.01, delays=0/0/0/0.01, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")
Jun 22 00:00:21 server postfix/qmgr[3872]: BEAFE1981C6: removed
Jun 22 00:00:22 server postfix/smtp[13998]: B81861981B9: to=<dave@poieodesign.com>, relay=aspmx.l.google.com[74.125.140.26]:25, delay=0.53, delays=0/0/0.01/0.51, dsn=2.0.0, status=sent (250 2.0.0 OK 1371873622 v32si4453917yhp.356 - gsmtp)
Jun 22 00:01:01 server /USR/SBIN/CRON[15466]: (root) CMD (   bash /var/xdrago/guest-water.sh >/dev/null 2>&1)
Jun 22 00:01:01 server /USR/SBIN/CRON[15467]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:01:01 server /USR/SBIN/CRON[15468]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:01:01 server /USR/SBIN/CRON[15469]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:01:01 server /USR/SBIN/CRON[15470]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:01:01 server /USR/SBIN/CRON[15472]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:01:01 server /USR/SBIN/CRON[15471]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:02:01 server /usr/sbin/cron[8995]: (*system*csf_update) RELOAD (/etc/cron.d/csf_update)
Jun 22 00:02:01 server /USR/SBIN/CRON[16810]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:02:01 server /USR/SBIN/CRON[16811]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:02:01 server /USR/SBIN/CRON[16812]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:02:01 server /USR/SBIN/CRON[16813]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:02:01 server /USR/SBIN/CRON[16814]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:02:01 server /USR/SBIN/CRON[16819]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:03:01 server /USR/SBIN/CRON[17745]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:03:01 server /USR/SBIN/CRON[17746]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:03:01 server /USR/SBIN/CRON[17747]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:03:01 server /USR/SBIN/CRON[17748]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:03:01 server /USR/SBIN/CRON[17749]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:03:01 server /USR/SBIN/CRON[17750]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:03:08 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=220.176.122.58 DST=66.228.61.132 LEN=52 TOS=0x08 PREC=0x40 TTL=51 ID=29830 DF PROTO=TCP SPT=20641 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 22 00:03:11 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=220.176.122.58 DST=66.228.61.132 LEN=52 TOS=0x08 PREC=0x40 TTL=51 ID=30043 DF PROTO=TCP SPT=20641 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 22 00:03:18 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=220.176.122.58 DST=66.228.61.132 LEN=52 TOS=0x08 PREC=0x40 TTL=51 ID=30484 DF PROTO=TCP SPT=20641 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 
Jun 22 00:03:41 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=202.137.156.84 DST=74.207.229.250 LEN=439 TOS=0x00 PREC=0x00 TTL=46 ID=0 DF PROTO=UDP SPT=21736 DPT=5060 LEN=419 
Jun 22 00:04:01 server /USR/SBIN/CRON[18637]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:04:01 server /USR/SBIN/CRON[18639]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:04:01 server /USR/SBIN/CRON[18638]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:04:01 server /USR/SBIN/CRON[18640]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:04:01 server /USR/SBIN/CRON[18641]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:04:01 server /USR/SBIN/CRON[18642]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:05:01 server /USR/SBIN/CRON[19760]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:05:01 server /USR/SBIN/CRON[19761]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:05:01 server /USR/SBIN/CRON[19762]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 22 00:05:01 server /USR/SBIN/CRON[19763]: (root) CMD (bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1)
Jun 22 00:05:01 server /USR/SBIN/CRON[19764]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:05:01 server /USR/SBIN/CRON[19769]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:05:01 server /USR/SBIN/CRON[19771]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:05:01 server /USR/SBIN/CRON[19765]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:05:18 server drupal: http://aegir.poieohosting.com|1371873918|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:19 server drupal: http://aegir.poieohosting.com|1371873919|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:20 server drupal: http://aegir.poieohosting.com|1371873920|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:21 server drupal: http://aegir.poieohosting.com|1371873921|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:22 server drupal: http://aegir.poieohosting.com|1371873922|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:22 server drupal: http://aegir.poieohosting.com|1371873922|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:23 server drupal: http://aegir.poieohosting.com|1371873923|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:24 server drupal: http://aegir.poieohosting.com|1371873924|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:25 server drupal: http://aegir.poieohosting.com|1371873925|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:26 server drupal: http://aegir.poieohosting.com|1371873926|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:26 server drupal: http://aegir.poieohosting.com|1371873926|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:05:27 server drupal: http://aegir.poieohosting.com|1371873927|hosting_civicrm_cron|127.0.0.1|http://aegir.poieohosting.com/index.php||0||CiviCRM cron run was sucessful.
Jun 22 00:06:01 server /USR/SBIN/CRON[21746]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:06:01 server /USR/SBIN/CRON[21747]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:06:01 server /USR/SBIN/CRON[21748]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:06:01 server /USR/SBIN/CRON[21749]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:06:01 server /USR/SBIN/CRON[21750]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:06:01 server /USR/SBIN/CRON[21751]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:06:20 server postfix/qmgr[3872]: 4265C19804D: from=<www-data@server.poieohosting.com>, size=2520, nrcpt=2 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: 79E93F542B5: from=<www-data@server.poieohosting.com>, size=5498, nrcpt=2 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: 0A8EF1981AB: from=<www-data@server.poieohosting.com>, size=5224, nrcpt=2 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: B143DF54425: from=<kadebiyi@yahoo.com>, size=5053, nrcpt=1 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: B81861981B9: from=<www-data@server.poieohosting.com>, size=4124, nrcpt=2 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: 5C5C21981BB: from=<www-data@server.poieohosting.com>, size=1194, nrcpt=1 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: 589A8198021: from=<www-data@server.poieohosting.com>, size=1206, nrcpt=1 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: DF734F54429: from=<kadebiyi@yahoo.com>, size=5055, nrcpt=1 (queue active)
Jun 22 00:06:20 server postfix/qmgr[3872]: D8FC4F54638: from=<kadebiyi@yahoo.com>, size=5050, nrcpt=1 (queue active)
Jun 22 00:06:20 server postfix/smtp[22199]: DF734F54429: to=<s2paul1124@gmaill.com>, relay=none, delay=143198, delays=143198/0.04/0.01/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=gmaill.com type=MX: Host not found, try again)
Jun 22 00:06:20 server postfix/smtp[22196]: 589A8198021: host gmail-smtp-in.l.google.com[74.125.130.26] said: 450-4.2.1 The user you are trying to contact is receiving mail too quickly. 450-4.2.1 Please resend your message at a later time. If the user is able to 450-4.2.1 receive mail at that time, your message will be delivered. For more 450-4.2.1 information, please visit 450 4.2.1 http://support.google.com/mail/bin/answer.py?answer=6592 h49si4475298yhl.287 - gsmtp (in reply to RCPT TO command)
Jun 22 00:06:20 server postfix/smtp[22193]: 5C5C21981BB: to=<sally2@mytopwebhosting.com>, relay=mytopwebhosting.com[37.58.70.97]:25, delay=164114, delays=164114/0.03/0.32/0.12, dsn=4.0.0, status=deferred (host mytopwebhosting.com[37.58.70.97] said: 451 Temporary local problem - please try later (in reply to RCPT TO command))
Jun 22 00:06:20 server postfix/smtp[22196]: 589A8198021: to=<q.wer.t.y.u.iop.poiu.s.sz@gmail.com>, relay=alt1.gmail-smtp-in.l.google.com[173.194.75.26]:25, delay=46509, delays=46508/0.03/0.44/0.33, dsn=4.2.1, status=deferred (host alt1.gmail-smtp-in.l.google.com[173.194.75.26] said: 450-4.2.1 The user you are trying to contact is receiving mail too quickly. 450-4.2.1 Please resend your message at a later time. If the user is able to 450-4.2.1 receive mail at that time, your message will be delivered. For more 450-4.2.1 information, please visit 450 4.2.1 http://support.google.com/mail/bin/answer.py?answer=6592 nx5si2394546vcb.35 - gsmtp (in reply to RCPT TO command))
Jun 22 00:06:22 server postfix/smtp[22201]: D8FC4F54638: to=<kerah@iamrah.com>, relay=none, delay=143199, delays=143196/0.03/2.7/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=iamrah.com type=MX: Host not found, try again)
Jun 22 00:06:50 server postfix/smtp[22182]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 22 00:06:50 server postfix/smtp[22182]: 79E93F542B5: to=<support@example.com>, relay=none, delay=151583, delays=151553/0.02/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:06:50 server postfix/smtp[22181]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 22 00:06:50 server postfix/smtp[22181]: 4265C19804D: to=<support@example.com>, relay=none, delay=406, delays=376/0.03/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:06:50 server postfix/smtp[22189]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 22 00:06:50 server postfix/smtp[22189]: B81861981B9: to=<support@example.com>, relay=none, delay=388, delays=358/0.02/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:06:50 server postfix/smtp[22183]: connect to example.com[192.0.43.10]:25: Connection timed out
Jun 22 00:06:50 server postfix/smtp[22183]: 0A8EF1981AB: to=<support@example.com>, relay=none, delay=302781, delays=302751/0.02/30/0, dsn=4.4.1, status=deferred (connect to example.com[192.0.43.10]:25: Connection timed out)
Jun 22 00:06:50 server postfix/smtp[22188]: connect to mail.rbatx.com[71.40.44.79]:25: Connection timed out
Jun 22 00:06:50 server postfix/smtp[22188]: B143DF54425: to=<hmorrison@rbatx.com>, relay=none, delay=143219, delays=143188/0.02/30/0, dsn=4.4.1, status=deferred (connect to mail.rbatx.com[71.40.44.79]:25: Connection timed out)
Jun 22 00:07:01 server /USR/SBIN/CRON[22708]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:07:01 server /USR/SBIN/CRON[22709]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:07:01 server /USR/SBIN/CRON[22710]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:07:01 server /USR/SBIN/CRON[22711]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:07:01 server /USR/SBIN/CRON[22712]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:07:01 server /USR/SBIN/CRON[22713]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:08:01 server /USR/SBIN/CRON[23622]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:08:01 server /USR/SBIN/CRON[23623]: (root) CMD (bash /var/xdrago/graceful.sh >/dev/null 2>&1)
Jun 22 00:08:01 server /USR/SBIN/CRON[23624]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:08:01 server /USR/SBIN/CRON[23625]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:08:01 server /USR/SBIN/CRON[23628]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:08:01 server /USR/SBIN/CRON[23626]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:08:01 server /USR/SBIN/CRON[23629]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:09:01 server /USR/SBIN/CRON[24334]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:09:01 server /USR/SBIN/CRON[24335]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:09:01 server /USR/SBIN/CRON[24336]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Jun 22 00:09:01 server /USR/SBIN/CRON[24337]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:09:01 server /USR/SBIN/CRON[24338]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:09:01 server /USR/SBIN/CRON[24339]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:09:01 server /USR/SBIN/CRON[24340]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:09:01 server /USR/SBIN/CRON[24329]: (CRON) error (grandchild #24336 failed with exit status 1)
Jun 22 00:09:29 server kernel: nginx: page allocation failure. order:1, mode:0x20
Jun 22 00:09:29 server kernel: Pid: 10010, comm: nginx Not tainted 2.6.39.1-linode34 #1
Jun 22 00:09:29 server kernel: Call Trace:
Jun 22 00:09:29 server kernel: [<c0189a30>] ? __alloc_pages_nodemask+0x530/0x6f0
Jun 22 00:09:29 server kernel: [<c013f67a>] ? mod_timer_pending+0xba/0x110
Jun 22 00:09:29 server kernel: [<c01afb13>] ? T.819+0xb3/0x2e0
Jun 22 00:09:29 server kernel: [<c01aff86>] ? cache_alloc_refill+0x246/0x290
Jun 22 00:09:29 server kernel: [<c0139826>] ? local_bh_enable+0x16/0x80
Jun 22 00:09:29 server kernel: [<c01b008d>] ? __kmalloc+0xbd/0xd0
Jun 22 00:09:29 server kernel: [<c050f07e>] ? pskb_expand_head+0x12e/0x200
Jun 22 00:09:29 server kernel: [<c050f5bd>] ? __pskb_pull_tail+0x4d/0x2b0
Jun 22 00:09:29 server kernel: [<c05d9263>] ? ipv4_confirm+0xd3/0x180
Jun 22 00:09:29 server kernel: [<c0517d6d>] ? dev_hard_start_xmit+0x1dd/0x3e0
Jun 22 00:09:29 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 22 00:09:29 server kernel: [<c059a900>] ? ip_finish_output2+0x260/0x260
Jun 22 00:09:29 server kernel: [<c052bcc2>] ? sch_direct_xmit+0xb2/0x170
Jun 22 00:09:29 server kernel: [<c0518069>] ? dev_queue_xmit+0xf9/0x320
Jun 22 00:09:29 server kernel: [<c059aa3b>] ? ip_finish_output+0x13b/0x300
Jun 22 00:09:29 server kernel: [<c059acaa>] ? ip_output+0xaa/0xe0
Jun 22 00:09:29 server kernel: [<c0599e78>] ? ip_local_out+0x18/0x20
Jun 22 00:09:29 server kernel: [<c059a257>] ? ip_queue_xmit+0x117/0x3d0
Jun 22 00:09:29 server kernel: [<c01892d9>] ? get_page_from_freelist+0x119/0x340
Jun 22 00:09:29 server kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Jun 22 00:09:29 server kernel: [<c01062c4>] ? check_events+0x8/0xc
Jun 22 00:09:29 server kernel: [<c01062bb>] ? xen_restore_fl_direct_reloc+0x4/0x4
Jun 22 00:09:29 server kernel: [<c01b004f>] ? __kmalloc+0x7f/0xd0
Jun 22 00:09:29 server kernel: [<c05ad70f>] ? tcp_transmit_skb+0x37f/0x660
Jun 22 00:09:29 server kernel: [<c05b0165>] ? tcp_write_xmit+0x1e5/0x4f0
Jun 22 00:09:29 server kernel: [<c05b04d4>] ? __tcp_push_pending_frames+0x24/0x90
Jun 22 00:09:29 server kernel: [<c05a3228>] ? tcp_sendmsg+0x7b8/0xaa0
Jun 22 00:09:29 server kernel: [<c0105ae7>] ? xen_force_evtchn_callback+0x17/0x30
Jun 22 00:09:29 server kernel: [<c011e0e5>] ? pvclock_clocksource_read+0xf5/0x190
Jun 22 00:09:29 server kernel: [<c05c1407>] ? inet_sendmsg+0x47/0xb0
Jun 22 00:09:29 server kernel: [<c044c105>] ? info_for_irq+0x5/0x20
Jun 22 00:09:29 server kernel: [<c044c750>] ? evtchn_from_irq+0x10/0x40
Jun 22 00:09:29 server kernel: [<c0506ed9>] ? sock_aio_write+0x129/0x180
Jun 22 00:09:29 server kernel: [<c0506db0>] ? brioctl_set+0x20/0x20
Jun 22 00:09:29 server kernel: [<c01b2886>] ? do_sync_readv_writev+0xb6/0xf0
Jun 22 00:09:29 server kernel: [<c01b2b06>] ? rw_verify_area+0x66/0x120
Jun 22 00:09:29 server kernel: [<c01b300a>] ? do_readv_writev+0xaa/0x1a0
Jun 22 00:09:29 server kernel: [<c0506db0>] ? brioctl_set+0x20/0x20
Jun 22 00:09:29 server kernel: [<c0173b21>] ? handle_percpu_irq+0x31/0x50
Jun 22 00:09:29 server kernel: [<c016ebc5>] ? audit_syscall_entry+0x1a5/0x1d0
Jun 22 00:09:29 server kernel: [<c01b313e>] ? vfs_writev+0x3e/0x60
Jun 22 00:09:29 server kernel: [<c01b3261>] ? sys_writev+0x41/0xa0
Jun 22 00:09:29 server kernel: [<c068fe51>] ? syscall_call+0x7/0xb
Jun 22 00:09:29 server kernel: [<c0680000>] ? sctp_rcv_ootb+0x50/0xf0
Jun 22 00:09:29 server kernel: Mem-Info:
Jun 22 00:09:29 server kernel: DMA per-cpu:
Jun 22 00:09:29 server kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    2: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    3: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    4: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    5: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    6: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: CPU    7: hi:    0, btch:   1 usd:   0
Jun 22 00:09:29 server kernel: Normal per-cpu:
Jun 22 00:09:29 server kernel: CPU    0: hi:  186, btch:  31 usd:  48
Jun 22 00:09:29 server kernel: CPU    1: hi:  186, btch:  31 usd: 162
Jun 22 00:09:29 server kernel: CPU    2: hi:  186, btch:  31 usd:  60
Jun 22 00:09:29 server kernel: CPU    3: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    4: hi:  186, btch:  31 usd:   1
Jun 22 00:09:29 server kernel: CPU    5: hi:  186, btch:  31 usd: 162
Jun 22 00:09:29 server kernel: CPU    6: hi:  186, btch:  31 usd:  56
Jun 22 00:09:29 server kernel: CPU    7: hi:  186, btch:  31 usd: 122
Jun 22 00:09:29 server kernel: HighMem per-cpu:
Jun 22 00:09:29 server kernel: CPU    0: hi:  186, btch:  31 usd:  17
Jun 22 00:09:29 server kernel: CPU    1: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    2: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    3: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    4: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    5: hi:  186, btch:  31 usd:  50
Jun 22 00:09:29 server kernel: CPU    6: hi:  186, btch:  31 usd:   0
Jun 22 00:09:29 server kernel: CPU    7: hi:  186, btch:  31 usd:  17
Jun 22 00:09:29 server kernel: active_anon:807532 inactive_anon:105721 isolated_anon:0
Jun 22 00:09:29 server kernel: active_file:261585 inactive_file:194049 isolated_file:0
Jun 22 00:09:29 server kernel: unevictable:0 dirty:2102 writeback:0 unstable:0
Jun 22 00:09:29 server kernel: free:653466 slab_reclaimable:33102 slab_unreclaimable:6182
Jun 22 00:09:29 server kernel: mapped:112174 shmem:172594 pagetables:8991 bounce:0
Jun 22 00:09:29 server kernel: DMA free:2052kB min:72kB low:88kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 22 00:09:29 server kernel: lowmem_reserve[]: 0 702 8112 8112
Jun 22 00:09:29 server kernel: Normal free:5376kB min:3352kB low:4188kB high:5028kB active_anon:0kB inactive_anon:0kB active_file:193764kB inactive_file:252432kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:719320kB mlocked:0kB dirty:4168kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:132408kB slab_unreclaimable:24728kB kernel_stack:1776kB pagetables:35964kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Jun 22 00:09:29 server kernel: lowmem_reserve[]: 0 0 59277 59277
Jun 22 00:09:29 server kernel: HighMem free:2606436kB min:512kB low:9356kB high:18200kB active_anon:3230128kB inactive_anon:422884kB active_file:852576kB inactive_file:523588kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7587492kB mlocked:0kB dirty:4240kB writeback:0kB mapped:448696kB shmem:690376kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jun 22 00:09:29 server kernel: lowmem_reserve[]: 0 0 0 0
Jun 22 00:09:29 server kernel: DMA: 5*4kB 4*8kB 5*16kB 2*32kB 5*64kB 2*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2052kB
Jun 22 00:09:29 server kernel: Normal: 1278*4kB 62*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5608kB
Jun 22 00:09:29 server kernel: HighMem: 42118*4kB 93676*8kB 53412*16kB 14939*32kB 3753*64kB 636*128kB 119*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 2606680kB
Jun 22 00:09:29 server kernel: 628118 total pagecache pages
Jun 22 00:09:29 server kernel: 0 pages in swap cache
Jun 22 00:09:29 server kernel: Swap cache stats: add 1561, delete 1561, find 1703/1704
Jun 22 00:09:29 server kernel: Free swap  = 1048572kB
Jun 22 00:09:29 server kernel: Total swap = 1048572kB
Jun 22 00:09:29 server kernel: 2097136 pages RAM
Jun 22 00:09:29 server kernel: 1911810 pages HighMem
Jun 22 00:09:29 server kernel: 21134 pages reserved
Jun 22 00:09:29 server kernel: 1789339 pages shared
Jun 22 00:09:29 server kernel: 1109276 pages non-shared
Jun 22 00:09:41 server mysqld: 130622  0:09:41 [Warning] Aborted connection 706870 to db: 'kiddieprepschool' user: 'kiddieprepschool' host: 'li170-191.members.linode.com' (Unknown error)
Jun 22 00:10:01 server /USR/SBIN/CRON[25053]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:10:01 server /USR/SBIN/CRON[25055]: (root) CMD (bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1)
Jun 22 00:10:01 server /USR/SBIN/CRON[25054]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:10:01 server /USR/SBIN/CRON[25056]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:10:01 server /USR/SBIN/CRON[25058]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:10:01 server /USR/SBIN/CRON[25059]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:10:01 server /USR/SBIN/CRON[25057]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:10:54 server postfix/pickup[21855]: 750EA1981C6: uid=33 from= <www-data>Jun 22 00:10:54 server postfix/cleanup[26554]: 750EA1981C6: message-id=<20130622041054.750EA1981C6@server.poieohosting.com>
Jun 22 00:10:54 server postfix/qmgr[3872]: 750EA1981C6: from=<www-data@server.poieohosting.com>, size=1187, nrcpt=1 (queue active)
Jun 22 00:10:55 server postfix/smtp[26556]: 750EA1981C6: to=<busseyzujy@hotmail.com>, relay=mx1.hotmail.com[65.54.188.94]:25, delay=0.63, delays=0.01/0.01/0.2/0.4, dsn=2.0.0, status=sent (250  <20130622041054.750EA1981C6@server.poieohosting.com> Queued mail for delivery)
Jun 22 00:10:55 server postfix/qmgr[3872]: 750EA1981C6: removed
Jun 22 00:11:01 server /USR/SBIN/CRON[26612]: (root) CMD (   bash /var/xdrago/guest-fire.sh >/dev/null 2>&1)
Jun 22 00:11:01 server /USR/SBIN/CRON[26613]: (scout) CMD (/var/lib/gems/1.8/bin/scout 9793294a-9bf8-4947-a092-9b0e042fcc64)
Jun 22 00:11:01 server /USR/SBIN/CRON[26614]: (root) CMD (bash /var/xdrago/runner.sh >/dev/null 2>&1)
Jun 22 00:11:01 server /USR/SBIN/CRON[26615]: (root) CMD (bash /var/xdrago/clear.sh >/dev/null 2>&1)
Jun 22 00:11:01 server /USR/SBIN/CRON[26616]: (root) CMD (bash /var/xdrago/minute.sh >/dev/null 2>&1)
Jun 22 00:11:01 server /USR/SBIN/CRON[26617]: (root) CMD (bash /var/xdrago/second.sh >/dev/null 2>&1)
Jun 22 00:11:01 server /USR/SBIN/CRON[26620]: (aegir) CMD (/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch )
Jun 22 00:11:02 server kernel: Adding 1048572k swap on /dev/xvdb.  Priority:-1 extents:1 across:1048572k SS
Jun 22 00:11:20 server postfix/qmgr[3872]: 6FF36D9401D: from=<kadebiyi@yahoo.com>, size=5125, nrcpt=1 (queue active)
Jun 22 00:11:20 server postfix/qmgr[3872]: AC5AB19804E: from=<www-data@server.poieohosting.com>, size=1190, nrcpt=1 (queue active)
Jun 22 00:11:20 server postfix/smtp[27037]: AC5AB19804E: to=<rose1@mytopwebhosting.com>, relay=mytopwebhosting.com[37.58.70.97]:25, delay=33952, delays=33951/0.01/0.31/0.12, dsn=4.0.0, status=deferred (host mytopwebhosting.com[37.58.70.97] said: 451 Temporary local problem - please try later (in reply to RCPT TO command))
Jun 22 00:11:24 server postfix/smtp[26556]: 6FF36D9401D: to=<debbie.eber@gemail.com>, relay=none, delay=327961, delays=327957/0/4.1/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=gemail.com type=MX: Host not found, try again)
Jun 22 00:11:29 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=61.158.164.51 DST=173.230.129.23 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:29 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=61.158.164.51 DST=173.230.130.191 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:29 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=61.158.164.51 DST=173.230.132.162 LEN=40 TOS=0x00 PREC=0x00 TTL=106 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:42 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=218.65.30.104 DST=74.207.226.29 LEN=40 TOS=0x08 PREC=0x40 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:42 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=218.65.30.104 DST=74.207.229.29 LEN=40 TOS=0x08 PREC=0x40 TTL=108 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:42 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=218.65.30.104 DST=74.207.229.40 LEN=40 TOS=0x08 PREC=0x00 TTL=103 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:42 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:c1:08:00 SRC=218.65.30.104 DST=74.207.229.250 LEN=40 TOS=0x08 PREC=0x00 TTL=103 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:42 server kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=f2:3c:91:ae:d0:60:84:78:ac:5a:19:41:08:00 SRC=218.65.30.104 DST=74.207.229.239 LEN=40 TOS=0x08 PREC=0x40 TTL=107 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
Jun 22 00:11:58 server mysqld: 130622  0:11:58 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)
Jun 22 00:11:58 server mysqld: 130622  0:11:58 [ERROR] mysqld: Out of memory (Needed 8386572 bytes)</debbie.eber@gemail.com></rose1@mytopwebhosting.com></www-data@server.poieohosting.com></kadebiyi@yahoo.com></busseyzujy@hotmail.com></www-data@server.poieohosting.com></www-data></c0680000></c068fe51></c01b3261></c01b313e></c016ebc5></c0173b21></c0506db0></c01b300a></c01b2b06></c01b2886></c0506db0></c0506ed9></c044c750></c044c105></c05c1407></c011e0e5></c0105ae7></c05a3228></c05b04d4></c05b0165></c05ad70f></c01b004f></c01062bb></c01062c4></c0105ae7></c01892d9></c059a257></c0599e78></c059acaa></c059aa3b></c0518069></c052bcc2></c059a900></c059a900></c0517d6d></c05d9263></c050f5bd></c050f07e></c01b008d></c0139826></c01aff86></c01afb13></c013f67a></c0189a30></hmorrison@rbatx.com></support@example.com></support@example.com></support@example.com></support@example.com></kerah@iamrah.com></q.wer.t.y.u.iop.poiu.s.sz@gmail.com></sally2@mytopwebhosting.com></s2paul1124@gmaill.com></kadebiyi@yahoo.com></kadebiyi@yahoo.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></kadebiyi@yahoo.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></www-data@server.poieohosting.com></dave@poieodesign.com></drupalcron></drupalcron@server.poieohosting.com></drupalcron@server.poieohosting.com></drupalcron></support@example.com></www-data@server.poieohosting.com></daf2ed4beaa389cd2504a4feebce87a6@portal.poieodesign.com></www-data></dave@poieodesign.com></drupalcron></drupalcron@server.poieohosting.com></drupalcron@server.poieohosting.com></drupalcron></support@example.com></www-data@server.poieohosting.com></www-data>

Linode Staff

First step in diagnosing potential kernel-related things: upgrade your kernel.

2.6.39.1-linode34 is end-of-life, and contains root exploits. You should move off that kernel ASAP. Log into the Linode Manager, click on the Linode, then the configuration profile, and select "Latest 32bit" from the drop-down, save and reboot.

Also: are you hitting the 32bit 3G process memory limit?

-Chris

The errors were present from 9:52 until 16:38 when I realized there was an issue and rebooted. So surely my stats graphs would show something if it were an actually shortage of memory.

I received this from Linode today. Not sure if it has merit or not but would explain why I'm not actually out of memory. However, this still doesn't explain why it's happening every week.:

> One of our system administrators saw your post on the forum, and has a suggestion. He believes that the messages you're seeing from MySQL are due to hitting the 3 GB per process memory limit in 32 bit configurations. His suggestion is to set up a 64 bit deployment, and copy everything over, to see if that resolves your issue. You can resize your current disk images to make room for a new deployment, and use the "Deploy a Linux Distribution" link on the Dashboard to deploy the 64 bit distribution. One thing to note, with MySQL, you can't directly copy over the tables from your current deployment, and have to use something like mysqldump to export the tables to SQL statements, and then use the dump file to reimport them into the new MySQL setup.

Thanks Chris!

I've just updated the kernel. I'll have to wait till this weekend to see if it made a difference.

It is happening every week because some job executes there - possibly via some "internal" scheduler of one of the systems you're running.

The logs you pasted show that you're getting out of memory errors already - it might be a good idea to look when they start happening, earlier on.

This looks like the "final struggle" area, when memory's depleted, you have a few MB left, and things are choking for it.

In this log snippet we can see a call to "/var/aegir/drush/drush.php '@hostmaster' hosting-dispatch" just before each OOM message, but as this is a job that runs every minute, it might be just the "straw breaking the camel's back". However, it also may be that on weekends, for some reason, this job takes a huge amount of memory - so it might be a good idea to test it somehow.

If the program running out of memory would be mysql, a process limit like that would be possible, but look at the logs.

Sometimes it's nginx, sometimes it's mysql, sometimes it's kernel swap daemon… I think it's a case of general exhaustion.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Log in to Reply
Community Code of Conduct