| Author |
Message |
bji
Joined: 27 Aug 2003
Posts: 182
|
| Posted: Mon Feb 09, 2004 4:32 am Post subject: host5 performance issues 1:25 am PST? |
|
|
My Linode on host5 is really really slow right now ... my load (which I was able to see after waiting for about 4 minutes for SSH to log me in) was well over 4, and yet my Linode wasn't really doing that much. I did notice that it was doing an updatedb but that the CPU load for that was very low. Also, host5's load is listed as low. I am guessing that the problem is being caused by lots of disk activity due to cron jobs firing late at night. Do system logs show an unusual amount of activity at that time? Is there anything that can be done to prevent this? Thanks!
BTW, it got better in the time it took me to write this message. I saw the extreme slowness from about 1:20 am PST to about 1:30 am PST. But it may have started before that ... |
|
| Back to top |
|
bji
Joined: 27 Aug 2003
Posts: 182
|
| Posted: Mon Feb 09, 2004 4:38 am Post subject: |
|
|
Actuallly I think I might have been misreading the host load from my members page. It is possible that the host load was listed as high before and not low as I previously reported; it's certainly high right now in any case.
My Linode's load is still listed at from 2.5 to 3.5, but is much more responsive than before. It's now almost 1:40 am PST ... |
|
| Back to top |
|
caker
Joined: 15 Apr 2003
Posts: 2392
Location: Galloway, NJ
|
| Posted: Mon Feb 09, 2004 4:41 am Post subject: |
|
|
This is a nightly occurrence on the first few hosts, since they have Linodes with distro images deployed before I removed from cron most of what causes the high disk contention (updatedb, etc). I did pause each Linode for a few moments to determine if it was a specific Linode thrashing.
I've been working on a 2.6 host kernel and am preparing to deploy it fairly soon. 2.6 has improvements to the I/O system, and I'm looking forward to getting it online.
-Chris |
|
| Back to top |
|
bji
Joined: 27 Aug 2003
Posts: 182
|
| Posted: Mon Feb 09, 2004 1:39 pm Post subject: |
|
|
Well in the spirit of trying to help mitigate this problem through cooperation of the various Linodes on host5, I'm posting my /etc/crontab listings for hourly, weekly, daily, etc here. If you have a Linode on host5 could you do the same so that we can all try to find times that do not coincide?
04,09,14,19,24,29,34,39,44,49,54,59 * * * * root run-parts /etc/cron.fiveminutely
07 * * * * root run-parts /etc/cron.hourly
17 4 * * * root run-parts /etc/cron.daily
27 4 * * 1 root run-parts /etc/cron.weekly
47 4 2 * * root run-parts /etc/cron.monthly
Notes:
I made my own "cron.fiveminutely" to run stuff every five minutes. I use this for webalizer.
My hourly times are on seven minutes after the hour.
My daily times are at 4:17 am eastern time.
My weekly times are at 4:27 am eastern time on Monday morning.
My monthy times are at 4:47 am eastern time on the second day of each month. |
|
| Back to top |
|
rhashimoto
Joined: 13 Aug 2003
Posts: 55
|
| Posted: Tue Feb 10, 2004 10:24 am Post subject: |
|
|
| I run backups at 10:04a, 10:12a, 2:04p, 2:12p, 6:04p, 6:12p, 10:04p, and 10:12p all Pacific Time. Unfortunately, these are initiated from my home machine which currently is not synced with a time server. I'll try to remedy that shortly. |
|
| Back to top |
|
mikegrb
Joined: 16 Oct 2003
Posts: 266
Location: Dr Wierd's Lab, South Jersey Shore
|
| Posted: Tue Feb 10, 2004 11:58 am Post subject: |
|
|
| Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed. |
|
| Back to top |
|
bji
Joined: 27 Aug 2003
Posts: 182
|
| Posted: Tue Feb 10, 2004 1:33 pm Post subject: |
|
|
mikegrb wrote: Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.
It is odd to have them in pairs like that, but there probably is a reason; maybe backing up a database before some operation is done to it and then backing it up afterwards too?
I don't think that the backups that this person is doing is putting an appreciable load on host5. I have never noticed performance problems, and no one else seems to be reporting them, at the times that this person is doing backups.
The real problem seems to be all of the updatedb cron jobs firing at the same time late at night (eastern time). Hopefully people who have left their updatedb cron jobs to their default times will see these messages and remedy this. |
|
| Back to top |
|
rhashimoto
Joined: 13 Aug 2003
Posts: 55
|
| Posted: Tue Feb 10, 2004 1:59 pm Post subject: |
|
|
mikegrb wrote: Why backups that frequently and why pairs 8 minutes apart? What backup method are you using? Even if it is incremental this puts quite a load on the server to check files and determine which have changed.
The paired backup operations go in different directions, server to home and home to server. I separated them just to preserve scheduling flexibility.
The backups use hard links and rsync, to maintain multiple snapshots of the other machine. They get issued every four hours because I don't want to lose more than four hours of data :-)!
As bji suggested, this should not be a substantial load. They take no more than a couple minutes each. |
|
| Back to top |
|
| |
