Linode.com Forum

[caker]

Host7 was rebooted this afternoon. The disk queue seemed to get wedged and essentially DOS the rest of the Linodes on that machine. I'm still looking into the cause.

-Chris

[caker]

UPDATE: I/O problems still on host7, so early this morning I've shutdown all the Linodes (the nice kind) and rebooted host7 into 2.4.23. I know this will help, since 2.4.23 has performed well on host9 and 10.

I'm discovering that much of this problem comes from over-loaded Linodes. Make sure you optimize for low-memory environments. Running mysql, apache, 50 cronologs, named, sendmail, etc, etc all at the same time with the defaults is NOT going to get you far (with me or performance) :wink:

-Chris

[Quik]

I always wonder what you mean when you say "the nice kind" of shutdown/reboot - what's up with that?

-Quik

[caker]

Quik wrote: I always wonder what you mean when you say "the nice kind" of shutdown/reboot - what's up with that?
These guys on host7 have had a hard time before migrating with host4 requiring a hard-reboot -- where the Linodes didn't have a chance to properly shut-down.

So, this is a nice kind of reboot, where the Linodes all shutdown properly, I reboot the host, and then boot the previously-running Linodes back up.

-Chris

[Quik]

caker wrote: Quik wrote: I always wonder what you mean when you say "the nice kind" of shutdown/reboot - what's up with that?
These guys on host7 have had a hard time before migrating with host4 requiring a hard-reboot -- where the Linodes didn't have a chance to properly shut-down.

So, this is a nice kind of reboot, where the Linodes all shutdown properly, I reboot the host, and then boot the previously-running Linodes back up.

-Chris

Got ya.

Nice speedy reply!

[tetranz]

I may have missed something obvious but how can we tell which host we are on?

[Quik]

tetranz wrote: I may have missed something obvious but how can we tell which host we are on?

It may be a roundabout way of doing it but:

Log into your Linode control panel
Click "SSH, Remote Console and Desktop Information"
About half way down the page you will see something like this:

ssh username@hostx.linode.com

Where hostx is your linode's host :)

[schof]

caker wrote: I'm discovering that much of this problem comes from over-loaded Linodes. Make sure you optimize for low-memory environments. Running mysql, apache, 50 cronologs, named, sendmail, etc, etc all at the same time with the defaults is NOT going to get you far (with me or performance)

I got a Linode to learn about Linux, not because I already know it. Some questions:

1) Does this statement apply to a Linode 128 as well as the 64?
2) How much stuff can we run on our Linode and get good performance? I'm running mysql, apache, sendmail, and named.
3) What defaults should we change? Will you publish a guide to getting good performance from a Linode?
4) Alternatively, will you start providing disk images with defaults set correctly?
5) Are the configuration changes and performance problems you're talking about related to processor speed, memory allocation, or both?

[aaron]

i've got a couple memory saving suggestions for linode users:

if you're using spamassassin, try using the spamc/spamd client/daemon setup instead of running spamassassin itself.

supposedly, if you're using spamc/spamd or spamassassin, try using bogofilter instead. supposedly much smaller footprint. i've not tried this yet...

set a low number of spare apache processes (MinSpareServers, MaxSpareServers), and set a lower MaxRequestsPerChild if your apache seems to leak like mine does.

[tetranz]

Perhaps this should be another thread but:

aaron wrote: i've got a couple memory saving suggestions for linode users:

if you're using spamassassin, try using the spamc/spamd client/daemon setup instead of running spamassassin itself.


It probably depends on the frequency of messages. When I run spamc / spamd my 'top' looks like this:

PID USER PRI NI SIZE SWAP RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
26577 root 4 0 19832 5692 13M 2356 S 0.0 23.6 0:01 0 spamd

That's 23% of the memory being used all the time. That seems like a lot. I'm still pretty much a Linux newbie so even after reading 'man top' I'm not quite sure what the 13M RSS really means or how the swapping and sleeping really works. I guess 13M is about 23% of my 64 Mb memory.

If I run Spamassassin directly, of course it uses nothing at all unless its processing a message. So, if you have relatively low mail frequency, I suspect this might be the better option. Isn't it a tradeoff between CPU cycles and memory? I'm happy to be clarified or convinced otherwise.

I'm running Apache, MySQL, Sendmail and Spamassassin on a Linode 64 without spamd and most of the time my free memory varies between about 5 and 20 Mb and about half my 128 Mb swap is used. Does that sound okay? Everything is working very nicely.

I've complied my own PHP with lots of features which I don't really need. My libphp4.so is almost 10 Mb. I guess that makes httpd use more memory. I should probably recompile a more minimalist version.

Cheers
Ross

[caker]

schof wrote: 1) Does this statement apply to a Linode 128 as well as the 64?
The capacity of a poorly tuned system applies to any size machine, regardless if its a Linode or a real box. But, you do have twice the potential to work with on a Linode 128 over the L64.

schof wrote: 2) How much stuff can we run on our Linode and get good performance? I'm running mysql, apache, sendmail, and named.

You could be running 1000 processes and still be fine, it's so hard to say. I believe even a generally tuned system running in a Linode 64 has very large capacity.

schof wrote: 3) What defaults should we change? Will you publish a guide to getting good performance from a Linode?

I only know of a few tricks atm (apache threads, and mysql buffer values), but we can start a tuning thread, or perhaps a forum.

schof wrote: 4) Alternatively, will you start providing disk images with defaults set correctly?

I probably won't change the defaults unless this becomes more of a wide-spread problem. My comment earlier should have read "...running all that stuff on very busy sites with the defaults is bad...", but it was rather late :)

schof wrote:
5) Are the configuration changes and performance problems you're talking about related to processor speed, memory allocation, or both?
The changes affect mostly the amount of total memory + swap consumed by applications, either by reducing the amount of RAM they apps acquire, or by reducing/putting limits on the number of conncurrent running threads.

-Chris

[kenny]

caker wrote: but we can start a tuning thread, or perhaps a forum.

I would love a performance forum and maybe a forum for getting with other users of your host, this way we could plan things such as "Kenny's cron.hourly runs at 37, Bob's is at 42..".
I think we could benfit from some rrd graphs of the host's cpu, disk activity, etc. That way I (and others) could reschedule things such as cron.daily to run at lower peak times. Stats on our linode's cpu usage, etc would be also be great.. something like "Your linode is averaging 20% of the host's cpu!" I'd like to know if I was causing problems for the other users before it resulted in Chris having to restart a host or something.

Kenny