linode crash
I've experienced a crash recently, the following was in my kernel.log and the machine was unresponsive.
Did anyone encounter this? I'm not sure if it's a bug and if I should post a support ticket.
Any help would be appreciated…
Jul 8 10:58:02 mrs kernel: k2908kB low:3632kB high:4360kB active:322612kB inactive:164144kB present:544768kB pages_scanned:91379837 all_unreclaimable? yes
Jul 8 10:58:03 mrs kernel: lowmem_reserve[]: 0 0 0 0
Jul 8 10:58:03 mrs kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Jul 8 10:58:03 mrs kernel: lowmem_reserve[]: 0 0 0 0
Jul 8 10:58:03 mrs kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 2212kB
Jul 8 10:58:03 mrs kernel: DMA32: empty
Jul 8 10:58:03 mrs kernel: Normal: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2640kB
Jul 8 10:58:03 mrs kernel: HighMem: empty
Jul 8 10:58:03 mrs kernel: Swap cache: add 308063, delete 308064, find 7040090/7070248, race 0+34
Jul 8 10:58:03 mrs kernel: Free swap = 0kB
Jul 8 10:58:03 mrs kernel: Total swap = 262136kB
Jul 8 10:58:03 mrs kernel: Free swap: 0kB
Jul 8 10:58:03 mrs kernel: 140288 pages of RAM
Jul 8 10:58:03 mrs kernel: 0 pages of HIGHMEM
Jul 8 10:58:03 mrs kernel: 3075 reserved pages
Jul 8 10:58:03 mrs kernel: 59921 pages shared
Jul 8 10:58:03 mrs kernel: 0 pages swap cached
Jul 8 10:58:03 mrs kernel: 0 pages dirty
Jul 8 10:58:03 mrs kernel: 0 pages writeback
Jul 8 10:58:03 mrs kernel: 2 pages mapped
Jul 8 10:58:03 mrs kernel: 4759 pages slab
Jul 8 10:58:03 mrs kernel: 2592 pages pagetables
Jul 8 10:58:03 mrs kernel: printk: 23751 messages suppressed.
Jul 8 10:58:03 mrs kernel: oom-killer: gfp_mask=0x201d2, order=0
Jul 8 10:58:05 mrs kernel: [<c014b252>] out_of_memory+0x132/0x160
Jul 8 10:58:05 mrs kernel: [<c014cba6>] __alloc_pages+0x276/0x2e0
Jul 8 10:58:05 mrs kernel: [<c014e23f>] __do_page_cache_readahead+0x13f/0x2d0
Jul 8 10:58:05 mrs kernel: [<c0145ef6>] __delayacct_blkio_end+0x46/0x50
Jul 8 10:58:05 mrs kernel: [<c04c05db>] __wait_on_bit_lock+0x5b/0x70
Jul 8 10:58:05 mrs kernel: [<c01475e0>] sync_page+0x0/0x40
Jul 8 10:58:05 mrs kernel: [<c01475d6>] __lock_page+0x56/0x60
Jul 8 10:58:05 mrs kernel: [<c014a5cc>] filemap_nopage+0x15c/0x3b0
Jul 8 10:58:05 mrs kernel: [<c0156874>] __handle_mm_fault+0x234/0x1670
Jul 8 10:58:05 mrs kernel: [<c0108e0b>] timer_interrupt+0x46b/0x730
Jul 8 10:58:05 mrs kernel: [<c011196a>] do_page_fault+0x7ba/0xc24
Jul 8 10:58:05 mrs kernel: [<c012782a>] run_timer_softirq+0x3a/0x1f0
Jul 8 10:58:05 mrs kernel: [<c0122934>] __do_softirq+0x94/0x130
Jul 8 10:58:05 mrs kernel: [<c01111b0>] do_page_fault+0x0/0xc24
Jul 8 10:58:05 mrs kernel: [<c010562b>] error_code+0x2b/0x30
Jul 8 10:58:05 mrs kernel: Mem-info:
Jul 8 10:58:05 mrs kernel: DMA per-cpu:
Jul 8 10:58:05 mrs kernel: cpu 0 hot: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 0 cold: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 1 hot: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 1 cold: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 2 hot: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 2 cold: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 3 hot: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: cpu 3 cold: high 0, batch 1 used:0
Jul 8 10:58:05 mrs kernel: DMA32 per-cpu: empty
Jul 8 10:58:05 mrs kernel: Normal per-cpu:
Jul 8 10:58:05 mrs kernel: cpu 0 hot: high 186, batch 31 used:24
Jul 8 10:58:05 mrs kernel: cpu 0 cold: high 62, batch 15 used:16
Jul 8 10:58:05 mrs kernel: cpu 1 hot: high 186, batch 31 used:184
Jul 8 10:58:05 mrs kernel: cpu 1 cold: high 62, batch 15 used:47
Jul 8 10:58:05 mrs kernel: cpu 2 hot: high 186, batch 31 used:62
Jul 8 10:58:05 mrs kernel: cpu 2 cold: high 62, batch 15 used:41
Jul 8 10:58:05 mrs kernel: cpu 3 hot: high 186, batch 31 used:59
Jul 8 10:58:05 mrs kernel: cpu 3 cold: high 62, batch 15 used:52
Jul 8 10:58:05 mrs kernel: HighMem per-cpu: empty
Jul 8 10:58:05 mrs kernel: Free pages: 4852kB (0kB HighMem)
Jul 8 10:58:05 mrs kernel: Active:81921 inactive:41427 dirty:0 writeback:0 unstable:0 free:1213 slab:4759 mapped:2 pagetables:2592
Jul 8 10:58:05 mrs kernel: DMA free:2212kB min:84kB low:104kB high:124kB active:5280kB inactive:1580kB present:16384kB pages_scanned:4568115 all_unreclaimable? yes
Jul 8 10:58:05 mrs kernel: lowmem_reserve[]: 0 0 532 532
Jul 8 10:58:05 mrs kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Jul 8 10:58:05 mrs kernel: lowmem_reserve[]: 0 0 532 532
Jul 8 10:58:05 mrs kernel: Normal free:2640kB min:2908kB low:3632kB high:4360kB active:322404kB inactive:164128kB present:544768kB pages_scanned:92544721 all_unreclaimable? yes</c010562b></c01111b0></c0122934></c012782a></c011196a></c0108e0b></c0156874></c014a5cc></c01475d6></c01475e0></c04c05db></c0145ef6></c014e23f></c014cba6></c014b252>
4 Replies
> Jul 8 10:58:03 mrs kernel: oom-killer: gfp_mask=0x201d2, order=0
Looks like you just ran out of memory. On a VPS you don't have quite the resources (RAM in this case) that you might be used to.
Have a search through these forums for more info, but if you're running Apache and/or *SQL, essentially you'll need to carefully tune/prune their respective config files, and/or replace Apache with something a bit more lightweight (e.g. Nginx or Lighttpd).
Actually I don't run anything resource intensive but from the log files it seems Apache was to blame, I've played a bit with the configuration files, probably that is it.
I was a bit worried because until now it had ran about a year uninterrupted.
@valy:
Thanks for the tips.
Actually I don't run anything resource intensive but from the log files it seems Apache was to blame, I've played a bit with the configuration files, probably that is it.
I was a bit worried because until now it had ran about a year uninterrupted.
My advice is to run munin on your linode, it might help investigating any crashes or deviancies before the crash.
