EGAD, load going over 12 on host34

cat /proc/swaps might be a good place to start:

Filename                     Type            Size    Used    Priority
/dev/ubd/1                      partition       525304  204996  -1

That's over 200M in swap!

-Chris

So, when I want to see host34's load I should look at the amount

of swap space used on my linode? That doesn't make sense to me.

I have been using the load average on the first line of output from

the top command, which can show high load factors even when my

apparent CPU usage is very low. I thought swap space was related

to memory usage - is that incorrect? I'm no Joe Linux so it could be.

James Phillips, newbie at large

http://zunzun.com

Swap usage means disk usage. Every time a page gets swapped back into (or out of) RAM, you're using I/O.

I/O is one of the scarce resources on Linode, much scarcer than CPU. Caker has implemented a token scheme to make I/O fair. You can check you statistics by examining /proc/io_status.

When a process on Linux is blocked waiting for I/O, it takes up a full 1.0 on the CPU usage. That's why, I believe, you're seeing high "CPU load".

"loadavg" and "top" inside your Linode give the load of your Linode itself. When you see a high loadavg, it usually means processes are blocked, waiting for the disk. Each process blocked adds 1 to loadavg.

Having so much in swap means you're over-extending the memory of your Linode itself, causing much of the processes you are running to swap out to disk.

You should investigate placing limits on Apache threads and memory usage, mysql's memory usage, etc.

-Chris

Now there's advice a Linode newbie like me can sink his teeth

into! Thanks for explaining it to me, that does make sense.

James Phillips

now ju-u-u-st a bit less of a newbie

@caker:

"loadavg" and "top" inside your Linode give the load of your Linode itself. When you see a high loadavg, it usually means processes are blocked, waiting for the disk. Each process blocked adds 1 to loadavg.

I've been watching my Linode now that I understand this. Very

interesting and enlightening, and I can rewrite some of the site

to take the I/O limiting into account with good effect, I think.

May I ask why token_max is set to 20000 on my linode? From

what I gather poking around on the forums, it is because of the

swap size on my linode - is that correct? If so, reducing swap

usage should bump this number up?

James Phillips

@zunzun:

May I ask why token_max is set to 20000 on my linode? From what I gather poking around on the forums, it is because of the swap size on my linode - is that correct? If so, reducing swap usage should bump this number up?

Low token_max is down to excessive disk i/o usage, so your problem has been swap usage, not swap size per se.

I don't think upward changes to tokenmax occur automatically. When I got into the situation you're in now, I had to raise a ticket to get caker to set my tokenmax back to the original value.

The defaults are tokenrefill=512 and tokenmax=400000.

My understanding is if you reboot you will get your io tokens back…

Really? I was in that situation recently and got Caker to bail me out. I didn't try rebooting, though; that would be much easier! If it works please post here.

This is true when you shutdown and reboot your linode you IO is nolonger being used so it would go back up. Chris CAN give you a higher IOMAX and ipRefill rate for temp. if you have a good reason he has for me in the past

Well, Linode support was certainly kind and increased token_max

when I asked in a support ticket. I was asked not to thrash the

hard drive on host34 else it would have to be reduced, and this

seems perfectly reasonable.

I am curious about Gentoo re-syncing and updating: my guess

is that the host would be least busy on a Sunday, so that would

be the best time for this. It would likely make for a good bit of

swap space usage though. Any recommendations?

James Phillips

@zunzun:

I am curious about Gentoo re-syncing and updating: my guess is that the host would be least busy on a Sunday, so that would be the best time for this. It would likely make for a good bit of

swap space usage though. Any recommendations?
I run Gentoo on a Linode 64 (stage 1 install, not caker's distro) and I have only run into trouble (too much swapping - tokenmax and tokenrefill reduced - Linode slowed to a crawl) when linking glibc. I still had all my other services running at the same time (a little light apache, postfix and courier-imap) so I was being a bit greedy. This has been discussed here, and the advice from that discussion was to change your /etc/make.conf: MAKEOPTS="-j1"

I'll try the "-j1" this weekend on my Gentoo sync and check it.

Thank you for the advice.

James Phillips

@zunzun:

It would likely make for a good bit of

swap space usage though. Any recommendations?

If possible, it's also a good idea with Gentoo to shutdown as many services as possible during updates. If you have a 64, you'll want at least 20-25MB of ram free before starting your updates. It's less of a problem on a 96 depending on what your running. Just remember, the more ram you have free, the less your Linode needs to dip into swap, and the less I/O it's actually going to do copying pages from ram to swap and back.

@zunzun:

I'll try the "-j1" this weekend on my Gentoo sync and check it.

I synced up today (Sunday), X and Python were updated. The

-j1 option does appear helpful, but it is still quite a lot of drive

access to compile these big boys from source. I used:

watch -n 1 cat /proc/io_status

to babysit my drive usage during the afternoon's sync-n-build.

James Phillips

http://zunzun.com