--- Day changed --- Log opened Mon Apr 10 23:59:04 2006 00:43 -!- Newsome [~sorenson@adsl-75-0-136-70.dsl.chcgil.sbcglobal.net] has quit [Quit: Linux: Now with employee pricing!] 01:03 -!- snorp is now known as snorp|out 01:14 < fo0bar> Timing cached reads: 2972 MB in 2.00 seconds = 1487.11 MB/sec 01:14 < fo0bar> Timing buffered disk reads: 304 MB in 3.01 seconds = 101.08 MB/sec 01:14 < fo0bar> that's, umm, not on a shared box ;) 01:19 < fo0bar> caker: BTW, the 9550sx seems to be rather kickass, comparitavely. on the same hardware, same disks (2x200GB RAID1, SATA1), the 8000 and 9000 arrays got about 53MB/s, and the 9550 got about 65MB/s 01:20 < fo0bar> I didn't have an opportunity to test more disks 01:21 < TheFirst> oh hey 642K! we're really moving up in the world 01:35 < caker> fo0bar: mind running a few tiobench passes on it? 01:36 < caker> I'm particularly interested in the random reads tests, with a decent number of threads running 01:36 < caker> I'm going to do some xen-vs-vanilla tests tomorrow on the box I have here 01:36 < caker> Random Reads 01:36 < caker> File Blk Num Avg Maximum Lat% Lat% CPU 01:36 < caker> Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff 01:36 < caker> ---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 01:37 < caker> 2.6.16-xen0 528 4096 1 0.81 0% 4.806 200.91 0.00000 0.00000 0 01:37 < caker> ^-- somehow, that doesn't seem right 01:40 < fo0bar> caker: remind me tomorrow 01:41 < fo0bar> it's not powered on now 01:41 < fo0bar> (and I can only put one card in at a time) 01:41 < TheFirst> caker: any ideas on how to fix the io issues yet? 02:16 < taupehat> damn, lol 02:16 < mikegrb> lolz 02:16 < taupehat> taupehat.com is looking a bit ragged 02:19 < TheFirst> uhh i think you forgot a few newlines 02:20 < TheFirst> and damnit i'm gonna stick that lolz up mikegrb's arse...canidate for most annoying script ever 02:27 < taupehat> weird 02:27 < taupehat> I haven't done any site coding on it at all 02:27 < taupehat> second time the php has started acting weird since I changed to xenode 02:28 < TheFirst> well xennode has been at the pinnacle of suck today 02:28 < taupehat> last time, a debian update fixed if of all things 02:29 < taupehat> sure does look like a file got corrupted, doesn't it? 02:29 < taupehat> that first line of gibberish 02:29 < taupehat> restarting apache fixed it 02:30 * TheFirst needs to find some time to see if he can use lighttpd instead of apache 02:30 < taupehat> guess the file was fine, but reading the file wasn't fine 02:30 < TheFirst> apache gobbles up ram way to fsckin fast 02:30 < taupehat> eh, my ram profiles out just fine 02:30 < taupehat> now that I'm on a bigger node 02:31 < TheFirst> i can barely (read: not really) afford the 80] 02:31 < taupehat> ahh 02:34 < linbot> New news from forums: Did Host 56 reboot again in Xen Public Beta 02:35 < taupehat> no 02:35 < taupehat> it did not =] 02:41 < taupehat> so work was funny today 02:41 < taupehat> I wanted to evaluate a new sophos setup against some tough malware 02:42 < taupehat> but the filtering proxy I setup blocked access to all the sites liable to contain drive-by downloads, vx, kazaa-ish crap, etc 02:46 < warewolf> man was that a stroll down memory lane. 02:46 < taupehat> ? 02:46 < warewolf> anyone here know of bashish (the shell theme program) and it's predecesor, bashprompt? 02:47 < warewolf> I'll take that as a no. 02:48 < warewolf> way back in the day, in august of 1999 I created a massively overdesigned theme for bashprompt called 'blight' 02:48 < warewolf> (it was "ported" into bashprompt, but was removed in a later release) 02:48 < warewolf> I made screencaptures of all the current bashprompt themes, and gave them to the guy who ran the bash.current.nu website 02:49 < warewolf> all the themes had "warewolf@harmanr" in the picture -- I had my 15 minutes of fame :) 02:49 < NonVolatileRambo> i've seen them 02:49 < NonVolatileRambo> eterms, right? 02:49 < NonVolatileRambo> on a seemingly random cycle 02:49 < warewolf> wasn't required 02:49 -!- TheFirst_ [gaveup@CPE-70-92-72-102.new.res.rr.com] has joined #linode-xenbeta 02:50 < warewolf> ETerms were sooo oldschool and a trick 02:50 < warewolf> now we have xcomposite and can actually have transparent windows :) 02:50 < warewolf> that whole esetroot viewport shit was funny 02:50 < NonVolatileRambo> so you can make your terminal illegible by placing it over another terminal! 02:50 < NonVolatileRambo> PROGRESS! 02:50 < warewolf> NonVolatileRambo++ 02:50 < warewolf> NonVolatileRambo++ 02:50 < TheFirst_> hmmm looks like the node died, yay 02:51 < warewolf> somebody stole the roll of paper towels off my desk 02:51 < warewolf> goddamnit I use those to blow my nose 02:52 < warewolf> (I have really, really bad allergy problems) 02:52 * warewolf steals bakc 02:52 < warewolf> back 02:53 < TheFirst_> well i could have dealt without that info 02:53 < warewolf> haha sorry 02:53 < TheFirst_> bleh working on a resume BITES 02:54 < warewolf> TheFirst_: yes, yes it does. 02:54 -!- taupehat [me@taupehat.com] has quit [Ping timeout: 480 seconds] 02:54 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has quit [Ping timeout: 480 seconds] 02:54 < TheFirst_> sucks worse than a 2 dollar wh....you know the rest 02:55 < warewolf> heh 02:55 < warewolf> quit using part of my favorite phrase 02:55 < warewolf> IT'S MINE 02:55 < warewolf> when services are flapping, I use the phrase "X server is going up and down like a two dollar whore" 02:57 < TheFirst_> no way i've used that phrase for years 03:16 -!- cherring [~dce932aa@webuser.linode.com] has joined #linode-xenbeta 03:17 < cherring> is anyone on 56 able to get to their node? 03:22 < TheFirst_> 56 is dead 03:23 < cherring> thats the impression that I had 03:23 < caker> dead? 03:23 < caker> huh... 03:23 < TheFirst_> --- host56.linode.com ping statistics --- 03:23 < TheFirst_> 1870 packets transmitted, 0 received, 100% packet loss, time 1869671ms 03:23 < cherring> can't do anything, can't even log into LISH 03:24 < cherring> I can't even reboot via the LPM 03:24 < TheFirst_> it hasn't exactly been alive all day 03:25 < cherring> mmmmm 03:26 < cherring> might go get a coffee then :) 03:28 < cherring> oh it's coming up now :D 03:28 < cherring> faster than earlier today too :) 03:29 < cherring> how long till it gets kicked in the nuts again 03:30 < TheFirst_> the length it takes for the thrashers to boot up 03:30 < cherring> hmmm hope Im not one of them 03:30 < cherring> my stuff ran perfectly on the UML node 03:33 < TheFirst_> linode also had the iotokens... 03:33 < cherring> :-| 03:34 < cherring> I know, mine was always full, whats the system on xen? 03:35 < TheFirst_> some kernel thing i think...cfq? 03:35 * TheFirst_ pokes caker for corrections 03:36 < cherring> so does that mean that I can't really use any swap space? 03:36 < TheFirst_> it means a better scheduler needs to be worked out 03:36 < TheFirst_> or problems with this one fixed 03:37 < cherring> right so chances are my nodes inner workings are fine? 03:37 < TheFirst_> here's a nice error for ya caker...after the restart a boot failed on "Linode is already running" ... lish showed it as running, control panel showed it as off 03:38 < cherring> hmmm 03:38 * cherring scratches his head 03:41 < cherring> oh well at least I am back up for now, will go write some code :-) 03:41 < TheFirst_> wish i could say the same 03:42 < cherring> well I'm trying to be optimistic :-) 03:42 < cherring> I migrated retty early and Xen was smokin, must have been very few people on the host at that point 03:43 < TheFirst_> same here 03:43 < TheFirst_> wasn't until a week ago or so when things started acting up 03:43 < TheFirst_> think caker said there were two people thrashing at that point...when he paused the nodes things would fly again 03:43 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has joined #linode-xenbeta 03:43 < cherring> when I was first on there it was burning up and I was drooling a bit lol 03:43 < mikegrb> lolz 03:44 < cherring> mike's laughing, I must be the one thrashing lol 03:44 < mikegrb> lolz 03:44 < cherring> oh no its true ........ 03:45 * TheFirst_ kicks mikegrb and his script square in the nut 03:46 < cherring> TheFirst_ and TheFirst? 03:46 -!- TheFirst_ [gaveup@CPE-70-92-72-102.new.res.rr.com] has quit [Quit: You're a bloody puppet!] 03:46 < cherring> who was TheSecond? 03:47 < TheFirst> bah 03:49 < cherring> :) 03:50 < cherring> how many other xen hosts are there? how are they travelling? 03:52 < TheFirst> only 56 afaik 03:54 < cherring> ah ok. Sweet O, well hopefully it will all settle down :) 03:59 -!- alnr [~alan@cpe-69-200-85-107.nyc.res.rr.com] has quit [Quit: leaving] 03:59 < TheFirst> already starting to lose the io 03:59 < TheFirst> Timing buffered disk reads: 12 MB in 3.12 seconds = 3.85 MB/sec 04:02 -!- cherring1 [~dce932aa@webuser.linode.com] has joined #linode-xenbeta 04:03 < cherring1> I hope it doesnt hurt me 04:03 -!- cherring [~dce932aa@webuser.linode.com] has quit [Quit: CGI:IRC (Ping timeout)] 04:05 < cherring1> my pages still load fast and I can actually send mail so all is ok for now :) 04:26 < cherring1> poor caker, does he ever get to sleep ..... 05:16 -!- cherring1 [~dce932aa@webuser.linode.com] has left #linode-xenbeta [] 09:08 -!- bendy24 [~scott@bendy.netop.oftc.net] has joined #linode-xenbeta 09:34 -!- snorp|out is now known as snorp 10:07 -!- tompuppy [~tom@220-132-117-169.HINET-IP.hinet.net] has joined #linode-xenbeta 10:18 -!- tompuppy [~tom@220-132-117-169.HINET-IP.hinet.net] has quit [Quit: Leaving] 10:22 -!- petert [~peter@130.57.22.69] has joined #linode-xenbeta 12:25 -!- taupehat [me@taupehat.com] has joined #linode-xenbeta 12:25 < taupehat> 09:25 -!- taupehat [me@taupehat.com] has left #linode-xenbeat [] 12:25 < taupehat> hehe 12:25 < taupehat> so my node didn't come back online with the host reboot 12:25 < taupehat> issued it a boot command at the web page and it came right up though 12:25 < petert> (mine either) 12:27 < TheFirst> same here 12:27 < taupehat> whine! 12:27 < taupehat> on the other hand 12:27 < taupehat> it feels VERY smooth 12:28 < TheFirst> taupehat: damn right it's a whine and i want some cheese with it too! :P 12:28 < TheFirst> yah things have gotten speedy again 12:28 < taupehat> w00t 12:28 < taupehat> it feels good 12:28 < taupehat> wonder what caker did to it this time 12:28 < TheFirst> Timing buffered disk reads: 102 MB in 3.06 seconds = 33.37 MB/sec 12:28 < TheFirst> few hours ago it was 1MB/s 12:29 < taupehat> Timing buffered disk reads: 72 MB in 3.19 seconds = 22.59 MB/sec 12:29 < taupehat> so yeah 12:29 < taupehat> not bad 12:30 < TheFirst> hey compared to the last 48h we're in heaven 12:31 < TheFirst> but i kinda wonder if it's not just that the thrasher nodes didnt boot up after the last reboot? 12:31 < taupehat> maybe the thrasher nodes need a good thrashing =] 12:32 < TheFirst> i'd be more than happy to participate in said thrashing 12:33 < TheFirst> i gotta wonder how people let things get to that point...at leat with the crowd linode seems to attract 12:33 < TheFirst> i mean seems like such a basic admin hurdle 12:40 < bendy24> my node never thrashed until xen 12:40 < bendy24> such a basic admin hurdle my ass 12:40 < bendy24> poor software 12:41 < TheFirst> basic thing an admin should look for 12:41 < TheFirst> didn't say fixing it was necc. easy 12:41 < TheFirst> ;) 12:41 < bendy24> trying to fit amavis and spamassassin on a single 80 is far from easy 12:42 < TheFirst> it's not gonna happen 12:42 < TheFirst> well prolly doable if email is all the box does... 12:42 < bendy24> yep 12:42 < TheFirst> but i mean people that think spamassassin, clamav, apache, mysql, php, etc can all be done on an 80 are nuts 12:42 < bendy24> ive got it down to 50 megs into swap 12:43 < TheFirst> i just disabled spamassassin and clamav and have no swap until apache starts chewing things up 12:44 < TheFirst> just do filtering through procmail on my home box 12:46 < taupehat> I did all of that successfully on an 80 except SA 12:46 < taupehat> spamassassin is a big fat pig and I roasted it 12:47 < taupehat> put in postgrey instead 12:47 < TheFirst> yah 12:47 < taupehat> works very nicely 12:47 < taupehat> much lighter footprint 12:47 < TheFirst> clamav started to chug things down for me too 12:47 * bendy24 googles 12:47 < TheFirst> same here 12:47 < TheFirst> oh bah postfix stuff 12:47 < taupehat> and yes, I ran php, apache, mysql, postfix, imap-ssl, etc 12:47 * TheFirst is using qmail 12:47 < taupehat> easy 12:47 < taupehat> d'oh 12:47 < taupehat> TheFirst: don't let mikegrb hear you say that 12:47 < taupehat> he _hates_ qmail 12:48 < TheFirst> right now i've got it setup to talk with some rbl's and that's good enough for now 12:48 < TheFirst> taupehat: why? 12:48 < taupehat> mainly the shithead that develops it 12:48 < taupehat> that, and backscatter spam 12:48 < TheFirst> well i am not a fan of his either 12:48 < TheFirst> but his software works 12:48 < taupehat> heh 12:48 < TheFirst> and once you understand it it's easy to use 12:49 * taupehat is fine with postfix 12:49 < TheFirst> exim/postfix/sendmail i've monkied with for a decade and still can't setup properly 12:49 < taupehat> heh 12:49 < taupehat> it's all what you spend enough time geeking on 12:49 < taupehat> I've heard people say qmail is a bitch to keep going 12:49 < taupehat> because it's far too easy to nuke it doing basic config changes 12:49 < bendy24> postfix is pretty easy 12:49 < bendy24> almost too easy 12:49 < taupehat> aye 12:49 < bendy24> like this one girl i knew 12:50 < taupehat> bendy24: yeah, she was fun 12:50 < bendy24> ;) 12:50 < taupehat> ok 12:50 * taupehat returns to work 12:50 * taupehat has kicked djb in the shins! 12:51 < TheFirst> qmail just makes sense to me...i mean with email any server is easy to break ... especially with how careful you have to be about setting them up nowadays 12:52 -!- NonVolat1leRambo [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has joined #linode-xenbeta 12:55 -!- NonVolatileRambo [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has quit [Read error: Connection reset by peer] 13:04 -!- NonVolat1leRambo is now known as Spads 13:06 < fo0bar> caker: any special flags I need for tiobench? 13:07 < caker> well, I'm still experimenting, but even just -t1 gives the results I pasted 14:02 < bendy24> host56 seems very well behaved today 14:02 < bendy24> has everyone booted up? 14:03 < snorp> bendy24: yeah it's working well for me today too 14:03 < snorp> I switched to ubuntu from fc2 last night 14:03 < TheFirst> the thrashers must not be awake yet :P 14:03 < bendy24> i wonder if everyone has booted back up 14:03 < snorp> I wonder if that helped.... 14:04 < petert> heh 14:04 < snorp> petert! 14:04 < petert> snorp: no suse? ;) 14:04 < snorp> um, no :) 14:05 * petert tells on snorp 14:05 < snorp> they don't offer suse :) 14:06 < TheFirst> rpm based bites 14:06 < TheFirst> all hail the deb 14:07 < petert> ok, i regret saying anything now 14:09 < TheFirst> looks like you're outnumbered :P 14:10 < petert> no, it looks like we're wasting our time 14:23 < gpd> I spoke to timster on #quantact and it seems they use the 'reboot if broken' method 14:23 < gpd> and dynamically allocate more memory to thrashers 14:24 < gpd> he seemed to think sedf in 3.0.2 would help a lot 14:24 < TheFirst> sounds M$ish 14:29 < Spads> ? 16:29 < caker> gpd: I've been using sedf -- it's got nothing to do with IO throughput 16:29 < caker> even reducing cpu using to almost nil on the thrashers doesn't seem to help 16:31 < gpd> so you've been using 3.0.2? 16:31 < caker> yeah, -unstable which is 3.0.2 16:32 < caker> sedf has been in for quite some time 16:32 < snorp> what's going on today then? 16:32 < snorp> thrashers are not booted up? 16:32 < caker> no thrashers 16:32 < caker> guess not 16:32 < caker> or they got the idea 16:32 < bendy24> :) 16:32 < caker> heh 16:33 < snorp> caker: you should just limit people's swap :) 16:33 < snorp> so the thrashers can just run out of memory 16:33 < caker> snorp: easily circumvented with a swapfile 16:33 < snorp> heh, yeah 16:33 < caker> dd if=/dev/null of=/swapfile bs=1M count=256 16:33 < caker> mkswap /swapfile 16:33 < caker> swapon /swapfile 16:33 < caker> ... 16:35 < TheFirst> i'm telling ya...most effective means is to let a large angry mob know who they are 16:35 < TheFirst> i'm sure one of those is not going to be hard to scrounge up ;) 16:36 < caker> host56:~# hdparm -t /dev/sda 16:36 < caker> /dev/sda: 16:36 < caker> Timing buffered disk reads: 166 MB in 3.01 seconds = 55.09 MB/sec 16:36 < caker> :) 16:36 < snorp> /dev/hda1: Timing buffered disk reads: 66 MB in 3.04 seconds = 21.72 MB/sec 16:37 < snorp> better than floppy speed :) 16:37 < snorp> 55 MB/s is just ridiculous 16:37 < TheFirst> yah been hovering around 20 today 16:38 < gpd> so do we await more fixes from Xen or is there caker magik to be had? 16:39 < Spads> I think there's going to be a new caker throttle installed for teh Xen 16:39 < caker> I dunno yet ... XenSource is coming out with something in Q3 called XenEnterprise, which calls for "total IO guarantees, per domain" 16:39 < Spads> oooh 16:39 < caker> more than likely that'll be part of Xen-standard 16:39 < Spads> much better 16:39 < Spads> QoS 16:39 < caker> but in the meantime, I'll be looking at ways to implement the limiter 16:39 < Spads> Mary Queen of Scotts 16:41 -!- bendy24 [~scott@bendy.netop.oftc.net] has quit [Quit: leaving] 16:44 < TheFirst> i think putting up with the thrashers until Q3 is a bit much to ask 17:44 < caker> heh .. someone in #xen just discovered the "guest sets ip to the default route, network go boom" problem 17:44 < caker> muhaha :) 17:44 < caker> do this on any competitors VM: ifconfig eth0 17:54 < gpd> almost worth setting up a unixshell account just to bring them crashing down :) 20:47 < taupehat> ahh 20:47 < taupehat> I see the thrashers on 56 have come back online 20:50 < TheFirst> oh yay 20:50 < taupehat> caker: can you hit them with the two-by-four with the nail sticking out? 20:51 < TheFirst> i swear there's some thrashers gonna get look mighty funny walking into the ER to get my foot removed from their ass 21:03 < caker> # hdparm -t /dev/sda 21:03 < caker> /dev/sda: 21:04 < caker> Timing buffered disk reads: 154 MB in 3.02 seconds = 50.92 MB/sec 21:04 < caker> still thrashing? 21:05 < TheFirst> Timing cached reads: 2244 MB in 2.00 seconds = 1121.23 MB/sec 21:05 < TheFirst> Timing buffered disk reads: 76 MB in 3.03 seconds = 25.06 MB/sec 21:05 < TheFirst> faster....not as fast as it has been 21:05 < caker> what's the fastest buffered reads you've seen from hdparm? 21:06 < TheFirst> 54 i think 21:06 < TheFirst> usually when there's no thrashing it's around 38 21:12 < TheFirst> Timing cached reads: 2460 MB in 2.13 seconds = 1152.66 MB/sec 21:12 < TheFirst> Timing buffered disk reads: 16 MB in 3.13 seconds = 5.12 MB/sec 21:12 < TheFirst> thrashers back 21:12 * caker thinks of tricky ways he can get info out of the VMs 21:13 < caker> sysrq is about the best method 21:13 < caker> I could add a custom sysrq that dumps /proc/meminfo to console 21:13 < caker> and force sysrqs enabled, and not be disabled (/proc/sys/kernel/sysrq) 21:14 < caker> hmm .. I wonder what dm-NNN maps to in /proc/diskstats 21:14 < caker> 'cause that data is changing 21:15 * taupehat supports caker in his exercise of grey matter 21:16 < caker> if I could set up a "hidden" console, that'd be even better -- that way, no futzing with active users in console while I'm running sysrqs 21:17 < taupehat> I suspect any attempt to create a hidden console is going to set off IDS alarms 21:17 < caker> a-hah 21:17 < caker> # ll /dev/mapper/vg1-taupehat--1970* 21:17 < caker> brw------- 1 root root 253, 79 2006-04-11 03:25 /dev/mapper/vg1-taupehat--19708 21:17 < caker> note the minor number 21:18 < caker> brw------- 1 root root 253, 80 2006-04-11 03:25 /dev/mapper/vg1-taupehat--19709 21:18 < taupehat> eh? 21:18 < caker> # grep -e dm-79 -e dm-80 /proc/diskstats 21:18 < caker> 253 79 dm-79 20402 0 273550 183570 72565 0 274564 295690 0 138650 479260 21:18 < caker> 253 80 dm-80 24 0 192 50 0 0 0 0 0 50 50 21:18 < caker> ^-- your IO statistics from the HOST :) 21:18 < caker> w00t 21:18 < taupehat> ahh 21:18 < caker> that rocks 21:19 < taupehat> so how am I doing? 21:19 < caker> dunno, I don't have a tool to monitor that yet 21:19 < taupehat> heh 21:19 < taupehat> soon though =] 21:21 < caker> also: 21:21 < caker> # cat /sys/block/dm-80/stat 21:21 < caker> 24 0 192 50 0 0 0 0 0 50 50 21:22 < caker> dunno what those values are .. they seem kinda small 21:23 < caker> I really wonder how the other Xen providers are coping 21:23 < taupehat> heh 21:24 < taupehat> alias/tor it up and join their support chans 21:24 < caker> I know that the other UML providers all use my token-limiter patch 21:24 < caker> right 21:24 < taupehat> lurkage 21:24 < taupehat> /bin/sh: topic: command not found 21:25 < caker> ? 21:25 < taupehat> <-- derr 21:27 < caker> wanna run a benchmark for me, for fun? :) 21:28 < taupehat> sure 21:28 < taupehat> lemme shell up and you tell me what you want to see 21:28 < taupehat> Timing buffered disk reads: 22 MB in 3.01 seconds = 7.30 MB/sec 21:28 < caker> wget http://members.dslextreme.com/users/andylee/unixbench-4.1.0-wht.tar.gz 21:29 < caker> tar xvfz unixbench-4.1.0-wht.tar.gz 21:29 < caker> cd unixbench-4.1.0-wht 21:29 < caker> make 21:29 < caker> it's the VPS perf thread from WHT 21:29 < caker> ./Run 21:29 < caker> oh, actually -- before you do that 21:29 < caker> run that hdparm test a few times until it's consistant 21:29 < caker> then I'll change the ioprio on your thread 21:29 < taupehat> k 21:30 < taupehat> it's decreasing 21:30 < taupehat> and now increasing 21:30 < taupehat> I don't think it can get consistent 21:30 < caker> just a rough baseline will be enough 21:31 < taupehat> heh 21:31 < taupehat> 3.12, 8.36, 5.17, 11.38, 6.98, 14.45 21:31 < caker> ok, one sec 21:31 < caker> ok, I reniced everyone's xvd thread from -5 to 20 21:31 < caker> try again 21:32 < caker> I'm still not certain nice is tied to prio, but we'll check 21:32 < taupehat> a bit better it seems 21:32 * taupehat runs 21:33 < taupehat> Flavor of UNIX is not known. 21:33 < taupehat> Please define FLAVOR in Run script 21:33 < caker> lemme know when to un-nice yours 21:33 < caker> oh, hold off on that for a sec? 21:33 < caker> # renice -10 -p 19639 19640 21:33 < caker> 19639: old priority 19, new priority -10 21:33 < caker> 19640: old priority 19, new priority -10 21:33 < caker> ^-- your two xvd threads .. they're not -10, with everyone else set to 20 21:33 < caker> try hdparm again? 21:34 < taupehat> faster 21:34 < taupehat> much faster 21:34 -!- snorp is now known as snorp|out 21:34 < caker> numbers? 21:35 < taupehat> 16.52, 19.30, 23.84... 21:35 < taupehat> continuing to increase 21:35 < caker> yeah .. dunno if it's that, or the thrasher stopped.. 21:35 < taupehat> d'oh 21:35 < taupehat> 10.09 21:35 < caker> yeah, I just ran it, too 21:35 < taupehat> 42.71 21:35 < caker> nicer 21:36 < taupehat> ok 21:36 < taupehat> it's staying near the mid 20s 21:36 < taupehat> ready? 21:36 < caker> go for it 21:37 < caker> I un-niced yours and TheFirst's xvd threads .. we'll see if you guys can tell next time someone's thrashing 21:37 < taupehat> lol 21:37 < mikegrb> lolz 21:37 < taupehat> ok 21:37 < taupehat> final score: 0.0 21:38 < caker> something break? 21:38 < taupehat> in other words, it didn't appear to work 21:38 < taupehat> TEST BASELINE RESULT INDEX 21:38 < taupehat> Dhrystone 2 using register variables 376783.7 0.0 0.0 21:38 * taupehat runs it on his gaming rig for comparison 21:38 * caker runs it on the host 21:39 < taupehat> heh, "Dhrystone" 21:48 < caker> sysstat++ 21:49 < caker> FINAL SCORE 160.8 <-- loaded host 21:49 < taupehat> nice 21:50 < taupehat> 140.6 <-- gaming rig 21:50 < caker> ok .. iostat makes it easy to identify the thrashers 21:50 < taupehat> it was fun to watch gkrellm while runing that test 21:50 < taupehat> the proc chart was steadily climbing 21:50 < taupehat> so 21:50 < taupehat> if you can use iostat 21:51 < taupehat> you should be able to script a graph 21:51 < caker> something that sorts based on avg-over-$time (like last minute) usage 21:51 < caker> then look up the minor number to get the domid/username 21:51 < taupehat> yep 21:52 < taupehat> also, maybe create one for the users 21:52 < taupehat> so we can look at a graph of our usage 21:52 < taupehat> identify hotspots where cron jobs need to be tweaked, etc 21:52 < caker> I don't think I'll ever publish graphs for the hosts 21:52 < taupehat> no no 21:53 < caker> too easy to misinterpret 21:53 < taupehat> heh 21:53 < taupehat> I'm talking about looking at my own host 21:53 < caker> but yeah -- lots of stuff we could be graphing for the nodes 21:53 < caker> can you tell there's a thrasher? 21:53 < taupehat> and seeing that over the past week I've been steady except Tuesday, and today I really thrashed at 10AM 21:53 < caker> well, it calmed down a bit just now 21:53 < taupehat> heh 21:53 < taupehat> I use the spacebar test right here to see if there's a thrasher 21:54 < caker> spacebar? 21:54 < taupehat> yep 21:54 < caker> in bash? 21:54 < taupehat> very cheap and not at all accurate 21:54 < caker> it hurts interactivity that badly? 21:54 < taupehat> yeah 21:54 < caker> gesus 21:54 < taupehat> I hold down my spacebar in irssi here, and if it moves smoothly, then I'm happy 21:54 < taupehat> the other test I do is ssh in 21:54 < taupehat> if it takes more than 5 seconds for the password prompt, then I'm thinking thrash 21:55 * taupehat is trying to remember which app does the graph generation from the cli... gpd? 21:57 < caker> xm sysrq taupehat m 21:57 < caker> :) 21:57 < taupehat> ? 21:57 < caker> Free swap = 99320kB 21:57 < caker> Total swap = 99320kB 21:57 < caker> Free swap: 99320kB 21:57 < caker> etc.. 21:57 < caker> look at dmesg 21:57 < taupehat> ahh 21:58 < taupehat> it's full of call traces 21:58 < taupehat> oh! 21:58 < caker> yeah, just did t 21:58 < caker> 't' - Will dump a list of current tasks and their information to your 21:58 < caker> console. 21:58 < taupehat> [] syscall_call+0x7/0xb 21:58 < caker> feel free to dmesg -c all that 21:59 < caker> 'b' - Will immediately reboot the system without syncing or unmounting 21:59 < caker> your disks. 21:59 < caker> ^-- useful 21:59 < caker> after these two, anyway: 21:59 < caker> 's' - Will attempt to sync all mounted filesystems. 21:59 < caker> 'u' - Will attempt to remount all mounted filesystems read-only. 21:59 < taupehat> heh 21:59 < gpd> taupehat: dstat? 22:00 < taupehat> b does not sound so much useful as an excellent way to entertain yourself watching fsck run right after turning off the phone 22:00 < gpd> saidar is mildly interesting from a console 22:00 < taupehat> so is quake3 22:00 < caker> curses-based program which displays live system statistics 22:00 < caker> huh 22:01 * caker installerates 22:01 * taupehat believes the thrasher exists 22:01 < caker> dstat is pretty cool 22:01 < taupehat> "We have met the thrasher, and he is us." 22:02 * caker thanks gpd 22:02 < taupehat> jeez 22:03 < taupehat> apt-get do-anything is dog slow right now 22:03 < taupehat> and now it isn't 22:04 < TheFirst> so someones snooping the nodes now eh? 22:06 * taupehat sells pictures of TheFirst putting is pajamas on to the snoop the internets site 22:07 < gpd> asert( so the caker UML io patches utilize something fundamental to UML) 22:08 < caker> gpd: yeah, the usermode part of UML 22:08 < caker> gpd: meaning, it's libc based, with hooks into UML's io_thread 22:09 < caker> gpd: but, check this out: http://www.theshore.net/~caker/xen/blktap/, specifically the blkdump.c example 22:09 < caker> perfect place for the limiter 22:10 * gpd looks 22:12 < gpd> seems like the place to be doing it (but I have no clue) 22:15 -!- cherring [~dce932aa@webuser.linode.com] has joined #linode-xenbeta 22:20 -!- cherring [~dce932aa@webuser.linode.com] has quit [Quit: CGI:IRC (Ping timeout)] 22:39 < gpd> caker: are you in discussion with Andrew Warfield? 22:40 < caker> gpd: nope .. why do you ask? 22:40 < gpd> he is the block tap author - he says: If you use it for anything, please let me know! 22:41 < caker> yeah .. haven't used it yet :) 22:45 < gpd> taupehat: did you see my comment above about quantat? 22:46 < gpd> you were wondering how they cope - it seems they don't ;) 22:46 < gpd> but he did say that all quantat hosts are now Xen 22:46 < caker> me? .. yeah 22:46 < caker> huh 22:46 < gpd> timster said he just reboots the host when io problems kick in! 22:46 < caker> For the short term, I'm going to deploy more UML hosts, but with LVM .. when Xen is ready, it'll just be a host reboot 22:46 < caker> geez 22:47 < gpd> he also said that 'thrashers' are 'detected' by monitoring host load 22:47 < gpd> and that he just increases their ram in the short term 22:47 < gpd> however, this didn't seem very satisfactory as it required a reboot by the user from their LPM! 22:47 < gpd> all sounds very sketchy to me 23:04 < TheFirst> seems an odd deterrant....oh hey i want more ram i'll start thrashing 23:06 < taupehat> hehe 23:06 < taupehat> damn, if only I'd known 23:06 < TheFirst> (which if that works here i'm thrashing! :P) 23:06 * taupehat would have signed up and setup open-xchange there 23:07 < TheFirst> haha 23:07 < taupehat> god 23:07 < taupehat> that app is a pig even on decent hardware 23:07 < TheFirst> maybe shortterm until limiter or something is working you could just notify the thrashers that they should look at optimizing or upgrading or something? 23:07 < TheFirst> horde is a bit laggy on my xenode 23:07 < TheFirst> but usable... 23:08 < TheFirst> most of that lag i attribute to the apache being the whore that it is 23:08 < taupehat> heh 23:08 < taupehat> dunno 23:08 < taupehat> I've run some mean things against apache on my box 23:08 < taupehat> and it's usually ok 23:08 < taupehat> what nails me to the wall, bleeding, is when spammers hit my blog with distributed spam attacks 23:08 < taupehat> mysql eats all the shit when that goes down 23:10 < TheFirst> 10222 www-data 15 0 19036 8908 2904 S 0.0 10.8 0:00.39 apache2 23:10 < TheFirst> 9505 www-data 16 0 18980 8868 2888 S 0.0 10.8 0:00.58 apache2 23:10 < TheFirst> 9405 www-data 15 0 18816 8648 2832 S 0.0 10.5 0:00.13 apache2 23:10 < TheFirst> 1188 root 16 0 15036 5992 4092 S 0.0 7.3 0:01.08 apache2 23:10 < TheFirst> fair amount of ram being chewed up there :/ 23:10 < TheFirst> need to mess with lighttpd ... see ifi cant use that instead 23:11 < taupehat> 10856 www-data 16 0 20144 9488 3436 S 0 7.7 0:02.70 apache 23:11 < taupehat> 12129 www-data 16 0 20200 9012 2920 S 0 7.3 0:00.70 apache 1195 postgrey 16 0 10504 7804 2428 S 0 6.3 0:00.12 postgrey 1580 me 15 0 9352 6008 3376 S 0 4.9 0:07.30 irssi 23:11 < taupehat> damn 23:11 < taupehat> that formatted wrong 23:12 < TheFirst> yah i had to throw mine in notepad to fix formatting 23:12 < taupehat> notepad? 23:13 < TheFirst> laptop runs windows...kinda have to since my uni is completly windows centric, you had them a makefile and they say "give me a visual studio project file, it's the standard" 23:13 < taupehat> wow 23:13 < taupehat> that'd be the uni I handed the snotty "eat my shorts, I'm going to a real academic institution" letter to 23:14 < taupehat> I mean really 23:14 < taupehat> VS files? 23:14 < TheFirst> yah i'd do that too if you'd be willing to pay tuition :P 23:14 < taupehat> tell me you aren't a CS major 23:14 < taupehat> which uni, anyhow? 23:14 < taupehat> since I have a kid to save up for college and all that 23:14 < TheFirst> yah i'm a cs major...but uni is pretty much a formality to get the paper...i've taught myself pretty much everything on my own 23:14 < taupehat> which uni 23:16 < taupehat> hey caker did you see my earlier grieving about how apache and php are occasionally garbaging? 23:17 < taupehat> it's unique to my experience and only started when I changed hosts 23:17 < taupehat> no idea whether it's a memory/xen problem or a disk problem, but restarting apache fixes --- Log closed Tue Apr 11 23:59:00 2006