--- Day changed --- Log opened Wed Apr 05 23:59:03 2006 03:31 < TheFirst> gah i swear that xen boxes problems infected my home box...or at least that's my story and i'm sticking to it 03:48 -!- FireRabbit [~FireRabbi@c-71-197-224-25.hsd1.or.comcast.net] has joined #linode-xenbeta 03:49 < FireRabbit> everyone on host56 down again? 03:53 -!- FireRabbit [~FireRabbi@c-71-197-224-25.hsd1.or.comcast.net] has quit [Remote host closed the connection] 08:01 < valen2> We apparently have I/O issues this morning 08:02 < valen2> ...and back to normal. Odd. 08:07 < valen2> Humm 08:07 < valen2> Looks like a cyclic problem. 09:20 < linbot> New news from forums: Linode Platform Manager in Feature Request/Bug Report 10:16 -!- bendy24 [~scott@bendy.netop.oftc.net] has joined #linode-xenbeta 10:16 < bendy24> caker: caker caker caker 11:37 -!- Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has quit [Remote host closed the connection] 11:51 -!- Breaker_1 [~Breaker@67-39-238-2.ded.ameritech.net] has joined #linode-xenbeta 11:52 < Breaker_1> So, I'm wondering, now that I'm on the beta xen server, would it be possible to get freebsd ? I'd be more than happy to test that 11:56 -!- Breaker_1_ [~Breaker@165.138.28.198] has joined #linode-xenbeta 11:57 < mikegrb> ha, not for quite some time 11:57 < mikegrb> neither can read/write to the others filesystems 11:59 < Breaker_1_> mikegrb, you answering my question? or did I miss something while I was disconnected? 11:59 < mikegrb> yes, your question 11:59 < Breaker_1_> neither can read/write to other filesystems <--- ? 12:00 < mikegrb> linux and freebsd 12:01 < Breaker_1_> I wonder how factual that is 12:01 < mikegrb> ? 12:01 < Breaker_1_> we run both freebsd and xen here at work 12:01 < Breaker_1_> well 12:01 < Breaker_1_> xen on linux 12:01 < Breaker_1_> I should try to throw freebsd on the xen box 12:03 -!- Breaker_1 [~Breaker@67-39-238-2.ded.ameritech.net] has quit [Ping timeout: 480 seconds] 12:03 -!- Breaker_1_ is now known as Breaker_1 12:08 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has quit [Ping timeout: 480 seconds] 12:12 < Breaker_1> yeah, it looks like freebsd works in xen3 12:12 < Breaker_1> all the articles I've found say everything works but sleep 12:12 < mikegrb> yes 12:12 < mikegrb> but linux and freebsd can't read/write each others filesystem 12:12 < mikegrb> which is the problem 12:13 < mikegrb> freebsd won't be available until that happens 12:17 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has joined #linode-xenbeta 12:20 < caker> wow .. Keir fixed the zombie domains/backend driver already 12:22 < mikegrb> caker: I called him, just for you 12:38 < linbot> New news from forums: Reboot: host56 (graceful) - fix for zombie domains in Xen Public Beta 13:01 -!- FireRabbit [~FireRabbi@c-71-197-224-25.hsd1.or.comcast.net] has joined #linode-xenbeta 13:02 < FireRabbit> hello, my linode will not start again. same error as before. 13:03 < bendy24> the host is being rebooted 13:03 < caker> !forums 13:03 < linbot> caker: Reboot: host56 (graceful) - fix for zombie domains in Xen Public Beta || Linode Platform Manager in Feature Request/Bug Report || Spamlists and exim4? in Email/SMTP Related Forum || What are sit0 and gre0? in Xen Public Beta (1 more message) 13:04 < FireRabbit> ah 13:04 < FireRabbit> alright thanks then 13:07 < bendy24> pardon my ignorance, but what do the mean by 'domains'? 13:08 < caker> domains are nodes in Xenspeak 13:08 < bendy24> a xen host? 13:08 < caker> including the host 13:08 < bendy24> ah 13:09 < tierra> it's really just a more luxurious name for your spacious VPS ;) 13:22 < bendy24> ok! 1:30! 13:23 * bendy24 hits the reset button 13:23 < FireRabbit> i got the same "failed to get domid" error just now 13:24 < bendy24> me too 13:24 < caker> as you should 13:29 < FireRabbit> oh 13:30 < tierra> yeah, your not supposed to actually use your node 13:30 < FireRabbit> yeah, it was foolish of me to do the xen thing.. i have a support request in to be migrated back to UML. 13:30 < caker> FireRabbit: see how things go after this reboot, first 13:31 < caker> if it breaks again, we'll move out off 13:31 < FireRabbit> okay. 13:31 < FireRabbit> lemme know when. 13:31 < bendy24> i figure this is good training for caker if i can thrash my node every once and a while 13:31 < caker> indeed 13:31 < bendy24> :) 13:32 < bendy24> my graphs show a massive surge in outgoing traffic 13:32 < caker> really? weird 13:32 < bendy24> i need my logs 13:32 < caker> yeah .. I'm curious too 13:33 < bendy24> # Transfer/Mo 50.0 GB 13:33 < bendy24> # Incoming: 295 MB 13:33 < bendy24> # Outgoing: 10.2 GB 13:33 < bendy24> :/ 13:33 < taupehat> yikes 13:33 < taupehat> eh caker I see there's a fix posted 13:34 < bendy24> looks like it started about 18:00 last night 13:35 < taupehat> bendy24: you hosting warez again? 13:36 < bendy24> no, midget pr0n 13:36 < bendy24> i blame /. 13:36 < taupehat> oh ok 13:46 < bendy24> good to go? 13:47 < caker> still shutting down 13:47 < bendy24> yikes 13:48 < caker> it's not in the best of states with this stupid driver bug 13:50 -!- taupehat [me@taupehat.com] has quit [Ping timeout: 480 seconds] 13:51 < bendy24> there goes taupehat 13:51 < bendy24> heh 13:52 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has quit [Ping timeout: 480 seconds] 13:55 -!- valen2b [~valen@adsl-70-238-134-49.dsl.stlsmo.sbcglobal.net] has joined #linode-xenbeta 13:55 < valen2b> How's the shutdown progressing? 13:56 < caker> they're booting now 13:56 < valen2b> Cool. 13:56 < valen2b> I'm assuming all the new driver patches are in this time? 13:57 -!- taupehat_ [~c6ed8603@webuser.linode.com] has joined #linode-xenbeta 13:57 < taupehat_> heh 13:57 < caker> yup 13:57 < taupehat_> hi 13:57 < valen2b> Hello. 13:57 < bendy24> taupehat_! 13:57 < taupehat_> that was amusing 13:57 < bendy24> hes back 13:57 -!- petert [~peter@130.57.22.69] has joined #linode-xenbeta 13:57 < bendy24> yes, we all laughed at your expense 13:57 < taupehat_> o rly? 13:58 < bendy24> no 13:58 < bendy24> just me 13:58 < bendy24> :) 13:58 < valen2b> The reason there are two of me is because I'm accessing the computer from a remote location... And I'm too lazy to fire up VNC to access my x-Chat session. :) 14:01 -!- taupehat [me@taupehat.com] has joined #linode-xenbeta 14:01 -!- taupehat_ [~c6ed8603@webuser.linode.com] has quit [Quit: ] 14:01 * taupehat muahahas 14:01 < bendy24> is it safe to boot the xennode now? 14:01 < caker> yup 14:01 < bendy24> here goes... 14:05 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has joined #linode-xenbeta 14:05 < valen2b> Oh. Awesome.. It's back! 14:05 -!- sednet [~51cf1045@webuser.linode.com] has joined #linode-xenbeta 14:06 < petert> heh, slowly.. 14:06 < sednet> Nice to know my downtime is helping the xen project :) 14:07 < bendy24> the queue is full it seems 14:07 < bendy24> :) 14:08 < taupehat> it is kind of neat to watch xen work itself out 14:08 < bendy24> it is? 14:08 < bendy24> :) 14:08 < taupehat> even if it does mean my server dies now and then 14:08 < FireRabbit> petert, yeah but at the same time... i miss my blog being online =) 14:09 < taupehat> heh 14:09 < taupehat> mine is 14:09 < taupehat> afaik 14:10 < valen2b> Performance issues still seem to be around. 14:10 < caker> they will while all the nodes are booting 14:10 < valen2b> Ah. 14:11 < valen2b> Lots of stuff hitting the disks I guess 14:13 < FireRabbit> curious, how many domains are on this box? 14:13 < caker> ~45, for testing 14:13 < caker> various Linode sizes... 14:14 < sednet> Thats far more load than a UML host.. 14:14 < caker> yes 14:14 < tierra> making sure to test the limits... heh 14:14 < FireRabbit> oh, less than 45 on the uml boxes? 14:14 < tierra> 40 at most on the 96s I believe 14:15 < bendy24> cmon cmon 14:16 < caker> freaking ebtables 14:16 < caker> can't they get their locking straight? 14:17 < valen2b> Problems? 14:17 < caker> bendy24: you *might* need to reboot, if networking isn't up correctly 14:17 < bendy24> yeah, im not getting any pings 14:17 < caker> well, check Lish first, make sure it's up totally 14:17 < bendy24> boot is still in progress... 14:17 < caker> nod 14:18 < sednet> 10 minute boot 14:18 < bendy24> ok 14:18 < caker> I'm not surprised 14:20 < valen2b> Well, at least I get the satisfaction of saying "Well sonny, I was there when they first started offering Xen nodes... And boy did we have our problems... But we got through them." 14:22 < caker> everything ok now? 14:22 < bendy24> i'll tell you in a few minutes :) 14:23 < FireRabbit> hmmmmmmm the site says its booted, cant ssh yet 14:23 < sednet> Same here 14:23 < caker> Lish? 14:23 < bendy24> sshd hasnt started yet 14:23 < FireRabbit> oh there we go... its still booting 14:24 < FireRabbit> wow this takes a long time :-x 14:24 < bendy24> you try booting 45 instances of linux at the same time :) 14:24 < sednet> Hmm. Stuck on starting ssh 14:24 < FireRabbit> hehe 14:24 < valen2b> mutt is still sorting my inbox... 14:26 < FireRabbit> starting ejabberd..... 14:26 < bendy24> starting amavisd... 14:26 < warewolf> ejabberd? 14:26 < bendy24> let the thrashing begin! 14:27 < warewolf> which jabber daemon is that? 14:27 < caker> heh .. ok, I'll be back in 1 hr 14:27 < FireRabbit> warewolf, that's what it's called 14:27 < valen2b> It's back. 14:27 < bendy24> caker: good luck 14:27 < FireRabbit> http://ejabberd.jabber.ru 14:27 < FireRabbit> caker, that's for all your work on this 14:27 < FireRabbit> i appreciate the customer service 14:29 < sednet> Finally finished booting 14:31 < FireRabbit> starting apache now... =) 14:32 < sednet> Maybe this host needs an EMC array on it. 14:32 < bendy24> starting amavisd... 14:32 < valen2b> Haha 14:32 < valen2b> With 45 terabytes of storage? 14:32 < FireRabbit> yay i have a login prompt! 14:33 < sednet> That would suit me 14:33 < valen2b> So each linode gets 1 TB each? :) 14:33 < sednet> You can get little clariion ones 14:33 < sednet> Very nice and easy to admin too 14:34 < valen2b> I think it's time to head out. 14:35 < valen2b> Good luck with Xen. Hopefully this patch fixes more things than it breaks. ;) 14:36 -!- valen2b [~valen@adsl-70-238-134-49.dsl.stlsmo.sbcglobal.net] has quit [Quit: Where did that controlling terminal go?] 14:36 < FireRabbit> hmm it booted but it seems to be too slow to do anything :( 14:36 < FireRabbit> and mysql failed to start...hmm.... 14:36 < bendy24> everyone is still recovering 14:36 < bendy24> give it a few minutes 14:36 < FireRabbit> ok 14:45 -!- brianrumburg [~18ddddaa@webuser.linode.com] has joined #linode-xenbeta 14:52 < FireRabbit> still cant log in 14:56 < bendy24> me either 14:57 < FireRabbit> hmm i managed to ssh in ... but it is extremely slow, it feels like the disk is the bottleneck, but its hard to say 14:58 < bendy24> yeah, everything is still coming up 14:59 < warewolf> GAH OH NO 15:00 < warewolf> er 15:05 < brianrumburg> yea, I'm getting some bad lag on host56, too. 15:05 < bendy24> real bad lag 15:06 < brianrumburg> I've been watching my vm boot for about 5 minutes. 15:06 < brianrumburg> just got an IP about a minute ago. 15:08 * bendy24 nods 15:14 -!- OvrLrd-Q [erwin@ip70-178-141-131.ks.ks.cox.net] has joined #linode-xenbeta 15:18 < OvrLrd-Q> is it possible to get xfered back to a normal linode? 15:18 < bendy24> yeah, submit a ticket 15:19 < OvrLrd-Q> k, cuz looks like Xen still has a few . . performance . . . issues :) 15:20 < bendy24> well, the host was just rebooted 15:20 < bendy24> give it some time 15:20 < OvrLrd-Q> ah, didn't know that :) 15:22 < bendy24> seems to be getting better... 15:23 < OvrLrd-Q> top - 14:23:08 up 1:06, 1 user, load average: 23.92, 12.85, 6.59 15:24 < mikegrb> OvrLrd-Q: cat /proc/swaps && vmstat 1 5 15:24 < mikegrb> paste the output 15:25 < OvrLrd-Q> http://pastebin.ca/48401 15:25 -!- marc_in_lux [~gergesm@cable-83.217.130.243.coditel.net] has joined #linode-xenbeta 15:26 < mikegrb> your io isn't very high there 15:26 < mikegrb> your swap is kind of big for having just rebooted though 15:28 < OvrLrd-Q> there we go, back down to like .9 15:29 < OvrLrd-Q> heh, was just going crazy since the reboot on tuesday night, forgot to fix lighttpd config which kept apache from starting, came home checked the site again and was like "Noooooooo" 15:30 < FireRabbit> my linode is 100% unusable at this point.... :( 15:31 < valen2> Hah 15:31 < OvrLrd-Q> and is it just me or is there alot more kernel threads in xen the UML 15:31 < valen2> I accidently killed DNS on my node once. 15:32 < iggy> OvrLrd-Q: think it's just you... I think there's less 15:32 < valen2> "So, let's fix this open DNS server problem... Let's see what happens when I disable recursive lookups." 15:32 < valen2> 6 hours later... 15:33 < valen2> "Wonder why nothing is resolving anymore." *looks up some stuff* "Doh. I disabled recursive lookups." 15:33 < OvrLrd-Q> i mean with UML i was averaging like 80 processes 15:33 < OvrLrd-Q> with xen i got 116 15:33 < valen2> Ended up making a "views" based configuration to solve that issue. 15:33 < valen2> Yea 15:33 < valen2> I see more kernel processes too. 15:34 < valen2> Or at least I think there is more... Didn't do an empirical study. :) 15:34 < sednet> Some threads like migration fork one per CPU 15:37 < mikegrb> yes 15:37 < mikegrb> and your xenodes are all smp 15:37 < OvrLrd-Q> bleh, my putty session froze 15:38 < Breaker_1> yeah 15:38 < Breaker_1> my server just completely stopped responding 15:38 < caker> that's me 15:39 < iggy> does it actually do any good to have the xenodes as smp? 15:39 < caker> yes 15:39 < caker> FireRabbit: around? 15:40 < FireRabbit> i am 15:40 < caker> cat /proc/swaps, and paste here, please 15:40 < FireRabbit> hm, i got disconnected 15:40 < FireRabbit> i will try to reconnect 15:41 < caker> FireRabbit: you're swap thrashing 15:41 < FireRabbit> $ cat /proc/swaps 15:41 < FireRabbit> Filename Type Size Used Priority/dev/hda2 partition 263160 116504 -1 15:42 < FireRabbit> hmmmm 15:42 < FireRabbit> ouch 15:42 < caker> FireRabbit: can you stop apache and/or mysql for a moment? 15:42 < FireRabbit> i can try. 15:43 < Breaker_1> is there any way I can migrate back to whatever node I was on before? I don't think my users are too happy that my server keeps splitting and dropping 15:44 < caker> Breaker_1: which host were you on, and can I just reset it back (without moving the filesystems)? 15:44 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has quit [Remote host closed the connection] 15:44 < Breaker_1> that's a good question.. I don't know 15:44 -!- TheFirst [gaveup@your.friendly.neighborhood.hellmouth.info] has joined #linode-xenbeta 15:45 < caker> OvrLrd-Q: looks like you're swap thrashing, too 15:45 < caker> OvrLrd-Q: cat /proc/swaps? 15:45 < OvrLrd-Q> i can't even login 15:45 < caker> try now 15:45 < caker> I had it paused 15:45 < bendy24> :/ 15:45 < FireRabbit> its hanging right now trying to run the apache init script to stop it 15:46 < valen2> killall -9 httpd ? 15:46 < OvrLrd-Q> Filename Type Size Used Priority 15:46 < OvrLrd-Q> /dev/hda2 partition 262136 28920 -1 15:46 < caker> hmm .. ok 15:46 < FireRabbit> i suppose thats an option 15:46 < FireRabbit> altho i can't control-c or anything it seems 15:47 < valen2> Humm 15:47 < valen2> SSH login is taking a long time. 15:47 < bendy24> ahh finally 15:47 < bendy24> i have net 15:48 < OvrLrd-Q> hurm there's no io_status anymore? 15:48 < valen2> Yep 15:49 < valen2> Xen doesn't have a token limiter 15:49 < caker> woah .. 15:49 < caker> [1]+ Segmentation fault /vbin/li-dispatchd.pl 15:49 < OvrLrd-Q> yay segfaulting perl scripts 15:49 < valen2> I assume that's not a good program to have sig 11? 15:50 < mikegrb> valen2: neither does uml ;) 15:50 < mikegrb> it's a linode thing (io_status) 15:50 < valen2> Yea 15:50 < valen2> But as far as I know of... There is no token limiter patch for Xen yet. :) 15:50 < mikegrb> right 15:50 < caker> yet... 15:51 < FireRabbit> caker, ok, i killed it i think (trying to look at the process list now ...waiting...) 15:51 -!- sednet [~51cf1045@webuser.linode.com] has quit [Quit: to sleep. perchance to dream] 15:51 < FireRabbit> k yeah its gone 15:52 < valen2> I need to weed processes sometime 15:52 < valen2> Half of these I could just kill 15:53 < bendy24> i cuold use some weed 15:53 < caker> hmm 15:53 < caker> Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC 15:53 < caker> ------------------------------------------------------------------------------ 15:53 < caker> u0 RAID-1 INITIALIZING 68 - 465.651 ON OFF OFF 15:53 < FireRabbit> if i open an application like vim it takes a while to start but then runs fairly fast.. it really seems to be disk access that is the slowest issue for me... 15:53 < caker> maybe that means it's only got one disk to nab reads off of until it finishes 15:53 -!- petert [~peter@130.57.22.69] has left #linode-xenbeta [whee] 15:53 < valen2> Maybe 15:54 < valen2> Initializing... Sounds like it's in the process of setting up 15:54 < caker> it's resyncing the disks 15:54 < brianrumburg> did you have a disk fail? 15:54 < valen2> I remember EverQuest used to have problems when disk drives failed... They ended up bringing the servers down so the mirror process would finish faster. 15:54 < caker> no, hard reset 15:57 < taupehat> so what's the scooty? =] 15:58 < valen2> Killing the fat cat tomcat server. ;) 15:59 < OvrLrd-Q> hmmmm fun, i got 4 zombie process and i wonder what they are 15:59 < taupehat> haha 15:59 < taupehat> who's running tomcat? 15:59 < OvrLrd-Q> aaaah, php4-cgi . . . strange 15:59 < FireRabbit> my node is running a bit faster now actually 15:59 < OvrLrd-Q> must be lighttpd 15:59 < caker> pausing all for a moment 15:59 < brianrumburg> yay! 16:00 < valen2> caker loves that "pause all" button. 16:00 < valen2> I'm convinced he has a big red button for it complete with flashing lights. 16:00 < OvrLrd-Q> the easy button 16:00 < OvrLrd-Q> ? 16:01 < valen2> Yea 16:01 < valen2> Like that 16:01 < OvrLrd-Q> heh 16:01 < OvrLrd-Q> i liked the commercial where it was raining printer cartirdges 16:01 < OvrLrd-Q> cuz the guy accidently sat on it 16:01 -!- taupehat_ [~c6ed8603@webuser.linode.com] has joined #linode-xenbeta 16:01 < valen2> Hah 16:01 < taupehat_> zoinks 16:01 < brianrumburg> show of hands... who actually has a staples easy button on their desk? 16:01 < valen2> All nodes paused taupehat 16:01 < taupehat_> eh caker, you doing paused? 16:01 < taupehat_> ahh 16:01 < mikegrb> when we have offices, we /will/ have a revolving red light that comes on when a host goes offline 16:01 < taupehat> haha 16:01 < taupehat_> what's the causus pausi? 16:02 < FireRabbit> mikegrb, :) 16:02 < valen2> Haha 16:02 < mikegrb> I've set a few up for various things 16:02 < mikegrb> they rule 16:02 < mikegrb> and a traffic light that changes with ping times 16:02 < taupehat> oh look 16:02 < taupehat> I'm back online =] 16:02 < OvrLrd-Q> should get a defcon sign 16:02 -!- taupehat_ [~c6ed8603@webuser.linode.com] has quit [Quit: ] 16:02 < valen2> Maybe you need a computer voice that says stuff like "Host!! Fifty!! Six!! Is!! Offline!!" on the PA system. 16:02 < OvrLrd-Q> and as a server becomes more overloaded it moves to defcon 1 16:02 < taupehat> heh 16:03 < taupehat> ergh 16:03 < taupehat> slowish again? 16:03 < valen2> It's taking forever to shutdown tomcat. :/ 16:03 * taupehat knows caker is just having way too much fun 16:03 < FireRabbit> it was working there for a moment taupehat 16:03 < FireRabbit> :_ 16:03 < FireRabbit> :) 16:03 < valen2> There we go. 16:03 < taupehat> valen2: apparently you didn't read my "what's the stupidest thing you've done to your linode?" thread 16:03 < taupehat> otherwise you wouldn't have attempted tomcat 16:03 < taupehat> =P 16:03 < valen2> LOL 16:03 < mikegrb> lolz 16:03 < FireRabbit> hhahaha 16:04 < valen2> Maybe they need to rename it to... 16:04 < valen2> "fatcat" 16:04 < taupehat> thrashcat 16:04 < valen2> Yea 16:04 < valen2> Doesn't work to well with spamassassin... 16:04 < taupehat> valen2: I hereby dedicate this thread to you: http://www.linode.com/forums/viewtopic.php?t=1648 16:05 < valen2> Spamassassin is a PITA too 16:05 < taupehat> ok 16:05 < valen2> Haha 16:05 * taupehat fiddles with his storage array 16:07 < valen2> Enterprise web apps on Linode 64? Haha 16:08 -!- OverlordQ [erwin@ip70-178-141-131.ks.ks.cox.net] has joined #linode-xenbeta 16:08 -!- marc_in_lux_ [~gergesm@cable-83.217.130.243.coditel.net] has joined #linode-xenbeta 16:09 -!- OvrLrd-Q [erwin@ip70-178-141-131.ks.ks.cox.net] has quit [Read error: Connection reset by peer] 16:09 -!- marc_in_lux [~gergesm@cable-83.217.130.243.coditel.net] has quit [Read error: Connection reset by peer] 16:09 -!- OverlordQ is now known as OvrLrd-Q 16:16 < bendy24> paused again? 16:16 < caker> no 16:16 < bendy24> ugh 16:17 < bendy24> there we go 16:18 < bendy24> 100s of emails to scan 16:18 < FireRabbit> caker, can you please move me back to wherever my node came from? no need to copy the filesystem back. 16:18 < caker> FireRabbit: ok, shutdown? 16:18 < FireRabbit> ok, its in the queue 16:19 < valen2> It's taking forever to uninstall this stuff. 16:19 < valen2> Doing a bit of spring cleaning now. 16:19 < caker> FireRabbit: also, log out of the website, please 16:19 < FireRabbit> okay. 16:19 < FireRabbit> so all my configuration/disks/etc. still exists on the other server, it's just not enabled? 16:22 < caker> FireRabbit: ok, you're back on host54 16:23 < FireRabbit> ok great, thanks 16:24 < caker> thanks for testing :)... 16:24 < bendy24> heh 16:24 < caker> I'm setting up host57 now. We'll get some of you guys off host56 16:24 < FireRabbit> heh, you're welcome too bad it didnt work out better 16:25 < valen2> Cool. 16:25 < valen2> Host56 just seems to be crumbling under all the weight. :) 16:25 < brianrumburg> how many vm's you got running on 56? 16:26 < valen2> I heard around 46 16:26 -!- pclissold [~pclissold@sassenheim.clissold.nl] has joined #linode-xenbeta 16:27 < brianrumburg> impressive. 16:28 < iggy> be more impressive if it was actually handling it 16:30 < brianrumburg> lol 16:30 < mikegrb> lolz 16:30 -!- marc_in_lux_ [~gergesm@cable-83.217.130.243.coditel.net] has quit [Quit: Leaving] 16:31 -!- pclissold [~pclissold@sassenheim.clissold.nl] has quit [Quit: Trillian (http://www.ceruleanstudios.com] 16:32 < bendy24> thrash thrash 16:34 -!- pclissold [~pclissold@sassenheim.clissold.nl] has joined #linode-xenbeta 16:36 < valen2> Are you thrashing again bendy? 16:37 < bendy24> amavis is trying to sort through 100's of messages 16:37 < valen2> Ah yes 16:37 < valen2> Amavis 16:37 < valen2> Another serial thrasher. 16:37 < bendy24> indeed 16:37 < valen2> We need a list for the Wiki. 16:38 < bendy24> i need more ram 16:38 < valen2> "Wanted dead or alive... Serial thrashers.... Amavis, Tomcat, Java, Spamassassin." 17:08 -!- Breaker_1 [~Breaker@165.138.28.198] has quit [Quit: Leaving] 17:12 -!- bendy24 [~scott@bendy.netop.oftc.net] has quit [Quit: leaving] 17:19 -!- iggy_ [~iggy@12.45.184.235] has joined #linode-xenbeta 17:19 -!- iggy [~iggy@12.45.184.235] has quit [Read error: Connection reset by peer] 17:40 < caker> who wants to migrate to host57 first? 17:41 < valen2> Humm 17:41 < valen2> Need a guena pig? 17:41 < caker> no, I already moved mine 17:41 < valen2> Hah 17:41 < taupehat> I have no name!@moya:~$ touch foo 17:41 < taupehat> touch: cannot touch `foo': Permission denied 17:41 < taupehat> whee! 17:42 < taupehat> I love ldap! 17:42 < valen2> Well. I'll give it a whirl I guess. 17:42 < valen2> Same specs at 56 I assume? 17:42 < caker> actually, it's 200Mhz x2 faster :) 17:42 < valen2> Sure 17:42 < valen2> I'll go for it. :) 17:42 < caker> valen2: ok, re(login) to the LPM and you'll see the migration 17:43 < valen2> Running shutdown and logging out. :) 17:43 < valen2> This is intra-datacenter, right? 17:43 < caker> Yes 17:44 < valen2> Shutdown running. 17:44 < pclissold> what's the thinking behind migrating to 57? 17:44 < valen2> Migrate some of the load off of 56 17:44 < caker> pclissold: get some people off host56. Also, host57's raid isn't scanning 17:45 < caker> my hope is that things improve once that's completed on host56 .. but still, that's a problem if we get an unexpected reboot on these boxes 17:45 < caker> and, I don't believe the perofrmance problems started with it in that condition (it was OK before my first reboot) 17:46 < pclissold> I can move if you want 17:46 < valen2> Ok.. 17:46 < valen2> It's copying disk images now. 17:47 < caker> pclissold: ok, I'll set yours up when he's done 17:47 < pclissold> ok 17:52 -!- iggy_ is now known as iggy 18:09 < brianrumburg> I'll move to 57. 18:09 < brianrumburg> need me to shutdown? 18:10 < caker> brianrumburg: in a bit -- pclissold is next 18:10 < caker> host57:/vbin# dd if=/dev/zero of=bigfile bs=1M count=512 18:10 < caker> 512+0 records in 18:10 < caker> 512+0 records out 18:10 < caker> 536870912 bytes (537 MB) copied, 10.3748 seconds, 51.7 MB/s 18:10 < caker> host57:/vbin# dd if=bigfile of=/dev/null 18:10 < caker> 1048576+0 records in 18:10 < caker> 1048576+0 records out 18:10 < caker> 536870912 bytes (537 MB) copied, 10.2038 seconds, 52.6 MB/s 18:10 < caker> :) 18:11 < brianrumburg> (c; nice 18:12 < warewolf> vbin? 18:12 < caker> virtual bin .. where all the host goodness happens 18:12 < warewolf> interesting 18:13 < brianrumburg> I'm getting ready to step away for a bit... Could you just send me a quick email when it happens? 18:13 < caker> brianrumburg: I'll ping you here -- that ok? 18:13 < caker> brianrumburg: or, do you want me to execute the migration when it's your turn? 18:14 < brianrumburg> laptop is leaving w/ me unfortunately... 18:14 < brianrumburg> Just go ahead and do it when it comes time... 18:14 < caker> ok 18:14 < brianrumburg> ill the box boot automatically after migration? 18:14 < brianrumburg> ^w 18:14 < caker> I'll put a boot into the queue 18:15 < brianrumburg> tnx caker. l8r4now. 18:15 < caker> l8r 18:15 < brianrumburg> damn I'm 1337 18:16 < caker> !leet brianrumburg is the leetest 18:16 < linbot> caker: br!4nrumburg !z +h3 133+35+ 18:16 < brianrumburg> o m g 18:16 < caker> !leet [weather 37211] 18:16 * brianrumburg bows 18:16 < caker> linbot: ping 18:16 < linbot> pong 18:16 < linbot> caker: 73mp3r@ur3: 77�F / 25�C | Hum!d!+y: 44% | Pr355ur3: 29.74!n / 1007hP4 | C0nd!+!0nz: 0v3rc45+ | W!nd D!r3c+!0n: S0u+h | W!nd Sp33d: 9mph / 15km/h; 70n!gh+ - M05+1y c10udy w!+h 4 30 p3rc3n+ ch4nc3 0f 5h0w3rz 4nd +hund3r5+0rmz. N0+ 4z c001. L0wz !n +h3 10w3r 60z. S0u+h w!ndz 10 +0 15 mph. 4z 0f 4:04 PM CD7 0n 4pr!1 6, 2006; 18:17 < OvrLrd-Q> ewwwwwwww, so hard . . . to read 18:17 < brianrumburg> lol 18:17 < mikegrb> lolz 18:18 < brianrumburg> later guys 18:24 < valen2> Wow 18:24 < valen2> 50 minutes later and it's still copying my disk image 18:24 < warewolf> ofcourse 18:25 < warewolf> it's going half way across the US, if not farhter 18:25 < valen2> Well 18:25 < valen2> That's very inefficient 18:25 < valen2> Since it's probably going like 20 ft in reality. 18:25 < caker> not even .. it's just that host56 is slammed 18:25 < OvrLrd-Q> yea was wondering about that heh :) 18:26 * OvrLrd-Q goes back to coding java 18:27 < pclissold> caker: Can you run my migration and boot the node for me, please. It's 12:30 here and I need to go to bed. 18:27 < caker> pclissold: no problem 18:28 < valen2> Migration of FileSystem 1 is done. 18:28 < valen2> 42 minutes 15 seconds later. 18:28 < caker> what's left? swap? 18:28 < valen2> Yep 18:28 < caker> ok 18:29 < valen2> 512 MB Swap 18:29 < caker> so .. 5 minutes? 18:29 < taupehat> ahh, life is good 18:29 < TheFirst> sort out any of the issues yet? 18:29 < valen2> Hopefully. :) 18:29 < pclissold> caker: thanks. node is shuuting down 18:29 < caker> TheFirst: not really .. trying to get some load off the host, and waiting for the raid to init (scan) 18:30 < taupehat> caker: the raid on 56 is still working up? 18:30 < caker> taupehat: hmm? 18:30 < TheFirst> resort to mideval law: thrash and get thrashed? 18:30 < taupehat> caker: you said the raid is initing 18:30 < caker> Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC 18:30 < caker> ------------------------------------------------------------------------------ 18:30 < caker> u0 RAID-1 INITIALIZING 70 - 465.651 ON OFF OFF 18:30 < caker> yeah, it goes awefully slow when the drives are in use 18:30 < valen2> Wow 18:30 < valen2> Up to 70 now. 18:30 < caker> I'm not sure that is even having an effect 18:31 < valen2> And yea. 18:31 < valen2> Raid is slow when in heavy use. 18:31 < caker> because it hauls ass if a few of the heavy-disk users are paused 18:31 < taupehat> heh 18:31 < taupehat> so what to do with the people running tomc- uhh, heavy disk users? 18:32 < valen2> We could do the 1776 method 18:32 < taupehat> what 18:32 < taupehat> toss them into Boston Harbor? 18:32 < valen2> I believe they called it "tar and feathering" 18:32 < valen2> That works too. 18:32 < TheFirst> yes, threat of physical violence is always effective 18:33 < TheFirst> and if not it's fun to carry out :P 18:33 < taupehat> it's particularly useful when the threat is carried out over the internet against anonymous people thousands of miles away 18:33 < valen2> I need to write a parody program... 18:33 < valen2> Called "Thrashmaster 5000" 18:34 < valen2> Migration done. 18:34 < taupehat> although I have seen a couple of fistfights actually happen due to smack-talking online 18:34 < valen2> Wooo 18:34 < valen2> Speedy. 18:34 < taupehat> the first I saw happened in about 1990 or 1991 =] 18:35 < TheFirst> hah 18:35 < TheFirst> yes, i've seen such too 18:36 < TheFirst> or someone betraying someone else in a game....that seems to produce violence too 18:37 < valen2> What type of game? 18:37 < taupehat> yeah 18:37 < taupehat> videogames 18:37 < TheFirst> tw2002 was the one i always saw people get into it over 18:37 < valen2> Ah 18:37 < valen2> Like team killing? 18:37 < TheFirst> modern times....umm wow? everquest? 18:37 < taupehat> I have a good friend whose long-term girlfriend left him for another guy in a gaming clan 18:37 < taupehat> was sort of classic in a way 18:38 < OvrLrd-Q> lol 18:38 < mikegrb> lolz 18:38 < valen2> Hah 18:38 < taupehat> but the fight in 199* was over trash-talking on trex 18:38 < TheFirst> taupehat: you know things are screwed up when there's 'athetic nerds' :P 18:38 < taupehat> and both the guys were HUGE 18:38 < taupehat> ex-military 18:38 < taupehat> was one helluva fight 18:39 < taupehat> heh 18:39 < taupehat> that was back when trashing someone online meant a few things: 18:39 < taupehat> 1) you really knew who the person was 18:39 < taupehat> 2) he also lived in San Jose 18:40 < valen2> Um 18:40 < caker> your momma wears combat boots 18:40 < taupehat> 3) your friends and his friends all went to the same Denny's at 3am 18:40 < valen2> caker. 18:40 < valen2> Problem! 18:40 < TheFirst> caker: your momma did but she left them at my place last night! :P 18:40 < caker> TheFirst: I know, afterwords she came to mine 18:40 < valen2> Can't locate Term/LinComplete.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /vbin/lish line 23. 18:40 < valen2> BEGIN failed--compilation aborted at /vbin/lish line 23. 18:40 < valen2> Connection to host57.linode.com closed. 18:40 < caker> valen2: ok .. one sec 18:40 < taupehat> oof! 18:41 < TheFirst> caker: well thanks for the nasty visual...now i wont be able to sleep! 18:41 < taupehat> lish would be a perl app 18:41 < OvrLrd-Q> why shouldn't it :) 18:42 * taupehat just likes flaming perl 18:42 < taupehat> http://www.taupehat.com/images/misc/perl.jpg 18:42 < taupehat> soon... 18:42 < TheFirst> bah flame a language worthy of flaming...like tcl! 18:42 < taupehat> hehe 18:42 < taupehat> http://www.taupehat.com/images/misc/burp.jpg 18:42 < valen2> Or Mr. Java 18:43 < taupehat> oy vey 18:43 < taupehat> java 18:43 < taupehat> yeah, flaming java is like preaching to the choir, though 18:43 < caker> valen2: fixed 18:43 < TheFirst> i dont mind java...if i'm not doing file i/o in it 18:43 < valen2> Cool 18:48 < caker> figures .. Kier checked in another blkback fix 18:48 < valen2> Restart soon? 18:48 < caker> no .. it just deals with the device connecting to the frontend 18:49 < caker> he more or less rewrote the damn thing since yesterday 18:49 < caker> valen2: everything OK on this new host? 18:50 < valen2> Um 18:50 < valen2> Working on some networking issues 18:50 < valen2> Rebooted to see if that helped 18:50 < OvrLrd-Q> hurm 18:50 < valen2> Hmm 18:50 < caker> valen2: ifconfig, route -n ? 18:51 < caker> (I can see your console, if you just run those) 18:51 < valen2> Waiting on it to start up. 18:51 < valen2> Apparently ntpdate is not liking it. 18:51 < valen2> There we go 18:51 < caker> or it's timing out since your net isn't configed right :) 18:52 < valen2> Both eth0 and eth0:0 are up 18:52 < caker> ok, I know what it is 18:52 < caker> give me a few 18:52 < valen2> Alright. :) 18:54 < caker> valen2: reboot, please 18:55 < taupehat> heh 18:55 < taupehat> sounds like it wasn't valen2's fault 18:55 < valen2> Rebooted from lish. :) 18:55 < caker> damn lusers 18:55 < caker> j/k :) 18:56 < caker> pingity ping 18:56 < valen2> Network seems to be back 18:59 < valen2> Everything seems ok. :) 18:59 < valen2> Network works, ssh works. :) 18:59 < valen2> Time is set on the host. 19:00 < taupehat> wewt 19:00 < valen2> Need anything else checked/ 19:06 -!- pclissold [~pclissold@sassenheim.clissold.nl] has left #linode-xenbeta [] 19:08 < caker> valen2: I think that's it 19:09 < valen2> Yay. 19:19 < valen2> Nice and speedy again. :) 19:19 < caker> weird thing is, so is host56 19:19 < caker> very odd 19:20 < valen2> Weird. 19:21 < valen2> Well. 19:21 < valen2> I'm headed out. 19:21 < valen2> Good luck with Xen. :) 19:42 < fo0bar> MAY THE XEN BE WITH YOU 19:50 < caker> fo0bar: know off hand what the 3ware card is doing in INITIALIZING state? (and don't say it's initializing) 19:56 < mikegrb> it's init'ing 19:57 < fo0bar> caker: did you power down uncleanly? 19:58 < fo0bar> if so, it's probably doing a consistency check 19:59 < fo0bar> (didn't you get a BBU as well?) 19:59 < caker> that's what I figured 19:59 < caker> yes, I did 19:59 < fo0bar> hmm 19:59 < caker> I've seen it rebuild (when I yanked disks) 19:59 < fo0bar> is it on? :) 19:59 < caker> Yes, tested, all that good stuff 19:59 < caker> Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC 20:00 < caker> ------------------------------------------------------------------------------ 20:00 < caker> u0 RAID-1 INITIALIZING 75 - 465.651 ON OFF OFF 20:00 < caker> Port Status Unit Size Blocks Serial 20:00 < caker> --------------------------------------------------------------- 20:00 < caker> p0 OK u0 465.76 GB 976773168 KRVN03ZAH33JUD 20:00 < caker> p1 NOT-PRESENT - - - - 20:00 < caker> p2 OK u0 465.76 GB 976773168 KRVN65ZAGRKBJF 20:00 < caker> p3 NOT-PRESENT - - - - 20:00 < caker> ~Name OnlineState BBUReady Status Volt Temp Hours LastCapTest 20:00 < caker> --------------------------------------------------------------------------- 20:00 < caker> bbu On Yes OK OK OK 255 22-Mar-2006 20:00 < caker> I expected a hard reset to something .. at least it doesn't rebuild on a hard reset 20:00 < caker> (like softraid) 20:00 < caker> s/to/to do/ 20:01 < caker> rebuilds hurt big time, since it turns off write cache until it's 2/3s done, or something like that 20:01 < fo0bar> I've seen 8000-series cards move to INITIALIZING after hard reset, but not all of the time 20:02 < caker> I've been contemplating raid10 for better reads 20:02 < fo0bar> it does seriously help 20:03 < caker> essentially doubling read throughput, right? 20:03 < fo0bar> Bonnie's block read on RAID10 across 6 disks SATA (not SATA2) yields over 120MB/s 20:03 < caker> I've been seeing about 50-60M/sec on simple read tests 20:03 < fo0bar> 4 disks is about 90 20:03 < caker> ok 20:04 < caker> threads is what seems to kill us 20:04 < fo0bar> yeah, 55 seems to be the sweet spot for RAID1 I've seen 20:04 < mikegrb> http://www.spectsoft.com/wiki/RaveManual/Hardware/ControllerBoard/InstallationGuide 20:06 < caker> I need to do some tests with that threaded disk benchmark tool (forget the name) 20:06 < caker> tbench, possibly 20:08 < mikegrb> About Initialization 20:08 < mikegrb> Some RAID levels must be initialized for best performance. (For specifics, 20:08 < mikegrb> see “Initialization of Different RAID Types” on page 120.) Initialization can 20:08 < mikegrb> take place before the units are put into service (foreground initialization) or 20:08 < mikegrb> while the units are being used (background initialization). 20:09 < mikegrb> Initialization makes parity information valid. Foreground initialization does 20:09 < mikegrb> this by simply writing zeroes to all the drives so that they all have the same 20:09 < mikegrb> values. Background initialization uses an algorithm to perform initialization 20:09 < mikegrb> and does not rewrite existing data. 20:09 < mikegrb> from the user's guide 20:09 < mikegrb> http://www.3ware.com/support/UserDocs/3ware9000UsrGuide.pdf 20:10 < mikegrb> "Auto Initialization After Power Failure 20:10 < mikegrb> The 3ware controller detects and handles power failures. This mechanism is 20:10 < mikegrb> intended to ensure that redundant units have consistent data and parity. When 20:10 < mikegrb> a redundant unit is unexpectedly shutdown, there is a possibility some data 20:10 < mikegrb> and parity may be inconsistent. For redundant units, if a unit or sub-unit is 20:10 < mikegrb> detected to have been shutdown uncleanly, the unit or sub-unit will change its 20:10 < mikegrb> mode to either ‘Initializing’ or ‘Verifying’. 20:11 < caker> hmm 20:11 < fo0bar> so, how does the BBU help then? 20:11 < caker> 128M cache == unwritten data 20:11 < fo0bar> oh duh 20:11 < caker> heh 20:11 < mikegrb> the bbu ensures cached data that hasn't been written yet is retained 20:11 < fo0bar> no power to the disks 20:11 < fo0bar> so it can't be written 20:11 < mikegrb> yea 20:12 < mikegrb> initialization makes sure that all of the data that was supposed to be written made it to the disk 20:12 < caker> odd, since it appears to scan the entire array 20:12 < caker> but whatever .. I don't think it affects performance 20:12 < caker> since it seems to pause whenever there's activity 20:13 < caker> I need to read up on what the storsave settings actually mean 20:13 < caker> [storsave= (9550SX only)] 20:13 < fo0bar> caker: I think there's a non-zero performance hit, since if you load 3DM2, you can set the init/verify priority 20:13 < caker> 3DM2 == tw_cli ? 20:14 < fo0bar> no, the webserver 20:14 < caker> hmm 20:14 < fo0bar> the settings may be in tw_cli as well 20:14 < caker> I haven't seen that yet 20:15 < fo0bar> but yes, I have noticed that if the array is in init/verify, and there is heavy disk activity, it takes FOREVER to finish, so I'm guessing the default value is sufficient 20:16 < caker> /cx set verify=enable|disable|<1..5> (enable|disable for 9000 series) 20:16 < caker> 1..5 <-- prios? 20:16 < fo0bar> I'm guessing so, since the web interface had 5 radio buttons 20:16 < caker> there's no mention of INIT anywhere in the help .. init = verify, I'm guessing 20:17 < fo0bar> I can't remember which is high/low though 20:18 < mikegrb> 9000-series controllers show separate settings for Rebuild/Migrate Rate and 20:18 < mikegrb> Verify Rate, as shown in Figure 21. The Rebuild/Migrate Rate also applies to 20:18 < mikegrb> initialization. Although the same rate is used for rebuilding, migrating, and 20:18 < mikegrb> initializing, migrating has the highest priority. 20:20 < mikegrb> The priority of rebuild versus I/O operations is specified with 1..5, where 1 is 20:20 < mikegrb> more resources and 5 the least. Setting the value to 1 gives maximum 20:20 < mikegrb> processing time to rebuilds rather than I/O. Setting the value to 5 gives 20:20 < mikegrb> maximum processing time to I/O rather than rebuilds. 20:21 < mikegrb> so 1 = finish quicker, poorer preformance 20:21 < mikegrb> 5 = dog slow, best preformance in the meantime 20:21 < fo0bar> ok, then I believe the default is 4 20:21 < mikegrb> bottom of page 45 in the CLI user's guide 20:22 * mikegrb hasn't slept since early yesterday afternoon 20:22 * mikegrb goes to bed 21:15 -!- Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has joined #linode-xenbeta 21:25 < caker> this might be helpful... 21:25 < caker> http://www.theshore.net/~caker/xen/blktap/ 21:25 < caker> er 21:25 < caker> http://www.theshore.net/~caker/xen/blktap/READ.ME.txt 21:26 < caker> This directory contains a library and set of example applications for 21:26 < caker> the block tap device. The block tap hooks into the split block device 21:26 < caker> interfaces above Xen allowing them to be extended. This extension can 21:26 < caker> be done in userspace with the help of a library. 21:26 < caker> i.e. token-bucket implementation 21:29 < taupehat> aha 21:29 < taupehat> so we're going to have io_tokens again? 21:29 < taupehat> that'll help warewolf a fair bit =] 21:30 < caker> I dunno .. I'm not sure if poor performance w/ 40 threads reading is normal life, or if there's something else I should be hunting 21:30 < caker> but, not having (good) control over a thrasher isn't going to cut it, I don't think 21:30 < caker> how's host56 feeling at the moment? 21:30 < taupehat> sluggish 21:30 < caker> that's what I expected 21:30 < taupehat> has been all day, since you asked 21:30 < taupehat> heh 21:31 < taupehat> are you forkbombing yourself on there? =] 21:31 < caker> it was fine an hour ago .. vmstat was zerod .. so, it's activity that someone's doing 21:31 < caker> no .. my node is off :) 21:31 < taupehat> hmm 21:31 < caker> host57 is happy and idle, btw 21:31 < taupehat> I wonder if there isn't a way to implement some sort of automatic hysterisis that kicks in 21:32 < taupehat> like if a host starts hitting the node hard enough, that node gets clamped down on until whatever process settles down 21:32 < taupehat> it is very much how io_tokens worked, I suppose 21:32 < caker> yeah -- that was possible with that io-watchdog that was planned, along with the controls of the token-limiter patch 21:32 < taupehat> heh 21:32 < caker> The way that would have worked, is that token_refill would be set sky-high (same as token_max, actually) 21:32 < caker> and only clamp down if certain conditions were met 21:33 < taupehat> right 21:33 < taupehat> although the implementation that exists in uml is pretty much allright as far as I can see 21:33 < caker> ie, host load, node's swap usage, node's avg io usage, etc .. all caclulated together 21:33 < taupehat> oh ah 21:33 < taupehat> that would be better then 21:33 < caker> Yeah, that's why I never extended token-limiter -- it does what we need to extend it in userspace 21:41 < caker> I almost wonder if swaps would be better as loop disk images, so the host can cache 21:41 < caker> even though currently, the "host" is just another domain, with only 256M 21:41 < caker> I think I can have it use all available, and when nodes start it'll balloon it down 21:56 < taupehat> shared swap? 21:57 < caker> no -- like UML now, all the images go through the hosts's cache 21:57 < caker> but using LVM, the VMs are connected directly to the devices, so the only cache is in the VMs themselves 21:59 < taupehat> VM or VG? 21:59 * taupehat is suffering from a bit of acronymitis 21:59 < caker> Virtuam Machine 21:59 < caker> Virtual, even 21:59 < taupehat> heh 21:59 < taupehat> maybe that's esperanto 21:59 < taupehat> virtuam machinam 22:40 < linbot> New news from forums: Reboot: host33 in System and Network Status 23:18 -!- brianrumburg [~18ddddaa@webuser.linode.com] has quit [Quit: CGI:IRC (Session timeout)] 23:25 -!- alnr [~alan@cpe-69-200-85-107.nyc.res.rr.com] has left #linode-xenbeta [] 23:26 -!- Newsome [~sorenson@adsl-75-0-136-70.dsl.chcgil.sbcglobal.net] has joined #linode-xenbeta 23:29 < taupehat> 20:29:40 up 9:33, 1 user, load average: 0.06, 0.03, 0.01 23:29 < taupehat> woohoo! 23:29 < taupehat> nine hundred and thirty-three days!1!!eleventy-one!11!! 23:32 < linbot> New news from forums: Cant receive mail from outside my domain... in Email/SMTP Related Forum 23:38 -!- Newsome [~sorenson@adsl-75-0-136-70.dsl.chcgil.sbcglobal.net] has left #linode-xenbeta [Linux: Now with employee pricing!] --- Log closed Thu Apr 06 23:59:00 2006