Huge burst on nodebalancer, oom on server, out of ideas

So we have two linodes running squid being fed by a nodebalancer and the squids are caching off a 4 gig linode running mediawiki/mysql/php etc.

Everything runs great, squid takes 95 percent of the load of serving pages.

But once a day on most days, not at the same time, the server goes into OOM and thrashing and has to be re-booted.

Looking at the nodebalancer incoming connections we see a huge traffic spike about 600-700 connections/second and then an immediate return to our normal 80-100.

There is nothing in the squid logs or apache logs abnormal during these periods.

Any ideas? I want to try and attribute it to a run away process somewhere, but what could trigger a traffic spike on the nodebalancer?

Any help is greatly appreciated!

2 Replies

Maybe daily log rotate/daily crons cause everything to get backed up. Check if it is the same time.

First step: tune your configurations (or upgrade) so the server will never go into an OOM condition.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct