Linode.com Forum

[nurikabe]

I have an Apache2/PHP5 installation running on a Linode 300. Everything is great.. except after PHP has been inactive for a minute or so. Once PHP has been idle, performance on the next web request goes from a millisecond reponse to several seconds. (Same behavior whether I have APC turned on or off..)

Has anyone else experienced this, and is there a workaround?

Thanks!

[Xan]

I've no idea why this is happening, but the quick and dirty workaround is a cron job that requests dynamic output from your Web server and pipes it to null once a minute.

..yuck!

[tierra]

Usually this is because you've filled up your RAM, and after a minute or so of inactivity, said processes might be swapped out for something else, and it adds a good amount of time to swap it back in when it's needed again. This is a fairly general answer though, and may not be the case at all, but you didn't give us much detail to really go on. Some statistics on system resources could be helpful: output of 'cat /proc/meminfo', io_limiter info ('cat /proc/io_status'), a few of the top lines from top, etc.

[nurikabe]

Thanks for the responses! I think I've risen to the level of my incompetence as far as sys admin goes.

Output below. top sorted by memory indicates that mysql is the largest memory hog. Memory is on the low-ish side. Perhaps I should cut the number of mysql processes?


free -m:

total used free shared buffers cached
Mem: 288 275 13 0 39 29
-/+ buffers/cache: 207 81
Swap: 64 9 55


cat /proc/io_status:

io_count=2732964 io_rate=4 io_tokens=400000 token_refill=512 token_max=400000

cat /proc/meminfo:

total: used: free: shared: buffers: cached:
Mem: 302915584 288636928 14278656 0 40415232 37089280
Swap: 68149248 10403840 57745408
MemTotal: 295816 kB
MemFree: 13944 kB
MemShared: 0 kB
Buffers: 39468 kB
Cached: 30412 kB
SwapCached: 5808 kB
Active: 36740 kB
Inactive: 38972 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 295816 kB
LowFree: 13944 kB
SwapTotal: 66552 kB
SwapFree: 56392 kB

top:

top - 06:25:14 up 7 days, 16:44, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 51 total, 1 running, 50 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Mem: 295816k total, 282396k used, 13420k free, 39780k buffers
Swap: 66552k total, 10160k used, 56392k free, 30460k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1106 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.84 mysqld
1122 mysql 8 0 25300 24m 4076 S 0.0 8.6 0:00.74 mysqld
1123 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld
1124 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.11 mysqld
1125 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.05 mysqld
1128 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld
1164 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.29 mysqld
1165 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.30 mysqld
1166 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld
1169 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:04.24 mysqld
1179 mysql 9 0 25300 24m 4076 S 0.0 8.6 0:00.00 mysqld
28062 nobody 9 0 17684 13m 6228 S 0.0 4.7 0:05.18 httpd
28078 nobody 9 0 17460 13m 6352 S 0.0 4.7 0:03.44 httpd
28111 nobody 9 0 17564 13m 6204 S 0.0 4.7 0:03.27 httpd
28063 nobody 9 0 17372 13m 6144 S 0.0 4.6 0:03.91 httpd
28106 nobody 9 0 17224 13m 6256 S 0.0 4.6 0:02.98 httpd
28105 nobody 9 0 17200 13m 6092 S 0.0 4.6 0:02.66 httpd
28061 nobody 9 0 17056 13m 6056 S 0.0 4.6 0:09.86 httpd
28065 nobody 9 0 17092 12m 5984 S 0.0 4.5 0:03.07 httpd
28064 nobody 9 0 16900 12m 5752 S 0.0 4.3 0:02.54 httpd
28060 nobody 9 0 15084 11m 4104 S 0.0 3.9 0:05.71 httpd
25937 root 8 0 9284 7992 7308 S 0.0 2.7 0:01.67 httpd
1155 ntp 9 0 3672 3672 2760 S 0.0 1.2 0:00.01 ntpd
1250 postfix 9 0 2468 2468 1368 S 0.0 0.8 0:00.01 qmgr

[pclissold]

Most of the memory shown as being used by MySQL will be shared between all the instances of the program.

I think that configuring MySQL and Apache to use less memory will only be marginally beneficial on a Linode 300 using a 2.6 kernel. The kernel will not necessarily stop swapping out parts of running applications even if you reduce memory usage below the size of physical memory - it may well create a bigger disk cache. Disk caching on a Linode is of questionable value because caching is also done by the host system kernel. Your Linode has ~10% of memory (30412 kB - more that MySQL or Apache are using) in use for disk caching and is swapping out parts of running applications when thet are idle.

My suggestion: adjust the kernel's 'swappiness' value. The swappiness defaults to 60, on a scale of 0 (prefer shrinking disk cache) to 100 (prefer swapping out inactive pages). Try reducing your swappiness value to between 0 and 20: Code: echo 10 > /proc/sys/vm/swappiness
This will encourage the kernel to shrink the disk buffers before swapping out applications.

[nurikabe]

pclissold wrote:
My suggestion: adjust the kernel's 'swappiness' value. The swappiness defaults to 60, on a scale of 0 (prefer shrinking disk cache) to 100 (prefer swapping out inactive pages). Try reducing your swappiness value to between 0 and 20: Code: echo 10 > /proc/sys/vm/swappiness
This will encourage the kernel to shrink the disk buffers before swapping out applications.

I'm looking at this now. This may be a silly question, but do I have to login via lish to write to the vm/ directory? Currently there is no swappiness file on my gentoo distribution and trying to write to it results in:

Code: localhost ~ # echo 10 > /proc/sys/vm/swappiness
bash: /proc/sys/vm/swappiness: No such file or directory

Thanks again for the help.

[pclissold]

nurikabe wrote: This may be a silly question, but do I have to login via lish to write to the vm/ directory?
No, you just have to be root.


nurikabe wrote: Currently there is no swappiness file on my gentoo distribution and trying to write to it results in:

Code: localhost ~ # echo 10 > /proc/sys/vm/swappiness
bash: /proc/sys/vm/swappiness: No such file or directory
What kernel are you using?

[nurikabe]

[quote="pclissold"]What kernel are you using?[/quote]

Twould be 2.4.29-linode39-1um.

[pclissold]

2.4.x kernels aren't tunable for swappiness - it was introduced in 2.6. It's probably worth your while to upgrade - the VM code in 2.6 is generally better. My feeling - no hard data though - is that I've had less memory/swap issues since switching to 2.6.

[nurikabe]

pclissold wrote: My feeling - no hard data though - is that I've had less memory/swap issues since switching to 2.6.

Okay, finally had a chance to do this. (I was gearing up to recompile the kernel myself.. and then remembered that I can just switch it from the Linode control panel. How I do love Linode.)

The result: with 2.6 and reduced swappiness, PHP seems to be much less groggy when awakened. There's still room for improvement, but we've gone from delays of dozens of seconds to three or four. Thanks again for all your help!

Evan

[anderiv]

nurikabe wrote: Okay, finally had a chance to do this. (I was gearing up to recompile the kernel myself.. and then remembered that I can just switch it from the Linode control panel. How I do love Linode.)
Evan - with linode, as with any VPS service, you're forced into using the kernels that are provided for you by the host. As such, trying to compile and run your own kernel is an excercise in futility.