--- Day changed --- Log opened Mon Mar 27 23:59:03 2006 00:02 -!- tompuppy [~tom@220-132-117-169.HINET-IP.hinet.net] has quit [Quit: Leaving] 00:15 -!- Newsome [~sorenson@adsl-75-0-136-70.dsl.chcgil.sbcglobal.net] has quit [Quit: Linux: Now with employee pricing!] 08:12 < vodka> g'day 08:13 < vodka> I was wondering wether my migration request got registered or not 08:13 < vodka> not wanting to pressure anything, just curious. 11:07 -!- andersen [~0fe38945@webuser.linode.com] has joined #linode-xenbeta 11:14 -!- andersen [~0fe38945@webuser.linode.com] has quit [Quit: CGI:IRC (Ping timeout)] 11:52 < caker> vodka: still around? 11:52 < vodka> yes 11:54 < caker> vodka: ok, ready to migrate? 11:54 < vodka> in a minute 11:54 < caker> all right .. here's what I suggest you do: df -h, and then resize your filesystem(s) so that the migration will take less time 11:55 < caker> ... then resize back up on The Other Side 11:55 < caker> because transferring 9gigs across datacenters would take a very, very long time .. cool? 11:56 < vodka> sure 11:59 -!- petert [~peter@snuvv.teichman.org] has joined #linode-xenbeta 12:04 -!- vodka_ [~knarf@ip-81-11-249-244.dsl.scarlet.be] has joined #linode-xenbeta 12:05 -!- vodka [~knarf@ip-81-11-234-198.dsl.scarlet.be] has quit [Read error: Connection reset by peer] 12:11 -!- vodka_ is now known as vodka 12:12 < vodka> hmm... ok, shrank everything to fit... going to press the magical migrate button now 12:15 < bendy24> that crazy button... 12:15 < bendy24> vodka: make sure you backup your swap partition! i lost mine :( 12:17 < Spads> I encrypt mine 12:17 < bendy24> oO 12:17 < Spads> to protect it from prying oO's 12:31 -!- vodka_ [~knarf@ip-81-11-196-18.dsl.scarlet.be] has joined #linode-xenbeta 12:32 -!- vodka [~knarf@ip-81-11-249-244.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 13:16 -!- vodka [~knarf@ip-81-11-242-34.dsl.scarlet.be] has joined #linode-xenbeta 13:17 -!- vodka_ [~knarf@ip-81-11-196-18.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 13:55 -!- vodka_ [~knarf@ip-81-11-217-61.dsl.scarlet.be] has joined #linode-xenbeta 13:56 -!- vodka [~knarf@ip-81-11-242-34.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 14:20 -!- marc_in_lux [~gergesm@cable-83.217.130.243.coditel.net] has joined #linode-xenbeta 14:21 -!- vodka [~knarf@ip-81-11-219-174.dsl.scarlet.be] has joined #linode-xenbeta 14:23 -!- vodka_ [~knarf@ip-81-11-217-61.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 15:00 * bendy24 aquires Spads's private key 15:00 < Spads> !! 15:00 < Spads> HOW??? 15:01 < bendy24> now i can decrypt your swap partition! 15:01 < Spads> NOES! 15:01 < bendy24> muwhahaha 15:32 -!- vodka_ [~knarf@ip-81-11-213-182.dsl.scarlet.be] has joined #linode-xenbeta 15:33 -!- vodka [~knarf@ip-81-11-219-174.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 15:45 -!- marc_in_lux [~gergesm@cable-83.217.130.243.coditel.net] has quit [Quit: Leaving] 17:16 -!- sonorous_ [~simon@grolsch.attenuate.org] has joined #linode-xenbeta 17:16 -!- bendy24_ [~slb@router.sbend1meg.1meg.golden.net] has joined #linode-xenbeta 17:16 -!- sonorous [~simon@grolsch.attenuate.org] has quit [jupiter.oftc.net quasar.oftc.net] 17:16 -!- phlaegel [~phlaegel@atdot.ca] has quit [jupiter.oftc.net quasar.oftc.net] 17:16 -!- encode [~encode@blah.i.hate.w1ndo.ws] has quit [jupiter.oftc.net quasar.oftc.net] 17:16 -!- bendy24 [~slb@bendy.netop.oftc.net] has quit [jupiter.oftc.net quasar.oftc.net] 17:16 -!- phlaegel [~phlaegel@atdot.ca] has joined #linode-xenbeta 17:16 -!- encode [~encode@blah.i.hate.w1ndo.ws] has joined #linode-xenbeta 17:24 -!- sonorous_ is now known as sonorous 17:56 -!- mode/#linode-xenbeta [+o mikegrb] by caker 18:38 -!- vodka [~knarf@ip-81-11-197-115.dsl.scarlet.be] has joined #linode-xenbeta 18:39 -!- vodka_ [~knarf@ip-81-11-213-182.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 19:35 -!- vodka_ [~knarf@ip-81-11-192-212.dsl.scarlet.be] has joined #linode-xenbeta 19:37 -!- vodka [~knarf@ip-81-11-197-115.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 20:38 -!- OvrLrd-Q [erwin@ip70-178-83-96.ks.ks.cox.net] has joined #linode-xenbeta 20:38 < OvrLrd-Q> how tricky 20:44 < OvrLrd-Q> YAY it mounted this time 20:44 < caker> heh .. how many tries? 20:44 < OvrLrd-Q> 4-5 20:44 < OvrLrd-Q> grr what's the command to disconnect the lish screen 20:44 < caker> more than usual 20:44 < caker> CTRL-A d 20:44 < OvrLrd-Q> there we go, was trying C-c d 20:44 < caker> CTRL-a d, rather 20:50 < caker> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- 20:50 < caker> r b swpd free buff cache si so bi bo in cs us sy id wa 20:50 < caker> 1 7 193320 2520 108 288 9332 9144 9424 9144 932 747 34 2 34 30 20:50 < caker> 1 8 192676 1708 108 324 9548 7968 9548 7968 1095 799 20 0 59 20 20:50 < caker> 2 8 193416 1892 108 328 8520 7948 8520 7948 1179 714 46 1 8 44 20:50 < caker> 1 8 192720 2220 108 316 7704 6228 7704 6228 900 650 18 0 65 17 20:50 < caker> ^-- node thrashing like all hell (mine) 20:50 < caker> can anybody tell? 20:51 < OvrLrd-Q> what's the command for that anyways 20:51 < caker> for what? 20:51 < OvrLrd-Q> vmstat nm 20:59 * mikegrb puts caker in time out 20:59 * caker runs eatmem on mikegrb 21:00 < Spads> vmstat 7, for when you have that not-so-fresh feeling 21:00 < caker> no one can tell that node is swap thrashing? 21:00 < caker> Spads: haha 21:01 < Spads> I actually had someone chastize me for talking publicly about vmstat 21:01 < Spads> like in a restaurant 21:01 < Spads> like, that's not appropriate dinner conversation! 21:01 < caker> yeah, what's the real brand/term/product's name? 21:01 < Spads> monostat 21:01 < caker> that's it 21:01 < Spads> it is "monostat 7" 21:01 < caker> too funny 21:01 < Spads> and I was talking about "vmstat 5" 21:01 < Spads> or something 21:05 < caker> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- 21:05 < caker> r b swpd free buff cache si so bi bo in cs us sy id wa 21:05 < caker> 13 10 437360 1724 100 300 8676 3024 8744 3024 531 501 14 7 65 14 21:05 < caker> 7 13 438448 2340 100 304 4516 4668 4516 4668 873 494 26 30 25 19 21:05 < caker> 1 13 438992 2124 100 412 3296 3072 3296 3072 481 305 22 13 49 15 21:06 < caker> thrash thrash 21:06 < caker> welp .. I'm satisfied 21:07 < caker> that's a hell of an improvement over UML+image files 21:11 < Spads> oh yeah? 21:11 < Spads> still thrashing? 21:11 < Spads> lemme see how my interactivity is 21:11 < caker> a single UML thrashing like that would cripple a host 21:11 < caker> ok, thrashing now 21:11 < caker> cripple a host to the point we wouldn't be ablet o ssh in 21:11 * Spads runs updatedb 21:11 < Spads> doodloo 21:11 < caker> hence, our screen session maintained to all the hosts 21:12 < Spads> haha 21:12 < Spads> ouch 21:12 < caker> yup 21:12 < Spads> Xen xen xen xen xen 21:12 < Spads> [nick@golgatem(~)] time sudo updatedb 21:12 < Spads> real 0m43.948s 21:12 < Spads> user 0m0.700s 21:12 < Spads> sys 0m0.030s 21:12 < Spads> :D 21:12 < caker> Spads: more more 21:12 < Spads> let's see about debian daturbases 21:13 < Spads> that's pretty taxing 21:13 < Spads> woohoo 21:13 < Spads> still punchy! 21:13 < caker> damb 21:13 < Spads> of course, I still have lots of ram 21:13 < caker> thrash it baby 21:13 < Spads> man, what can I even do to test now? 21:14 < caker> Spads: wget http://www.theshore.net/~caker/patches/utils/eatmem.c 21:14 < caker> Spads: gcc -o eatmem eatmcm.c 21:14 < Spads> haha 21:14 < Spads> oh wait 21:14 < Spads> I know 21:14 < Spads> lemme forkbomb 21:14 < caker> ./eatmem 21:14 < caker> usage: ./eatmem size[K|M|G] [num_of_threads] 21:15 < caker> ./eatmem 40M 10 & <-- example 21:15 < Spads> :(){:|:&};: 21:15 < caker> heh 21:15 < Spads> ready? 21:15 < caker> go 21:15 < Spads> oops 21:15 < Spads> [nick@golgatem(/tmp)] :(){ :|:& }; : 21:15 < Spads> [1] 3871 21:15 < Spads> there we go 21:15 < Spads> how are you doing? 21:16 < caker> still thrashing 21:16 < Spads> we need to see if a third one can do something taxing 21:16 < caker> third node? 21:16 < Spads> yeah 21:16 < caker> that forkbomb doesn't gen IO .. oh, i guess it would with thrashing 21:17 < Spads> yeah, it thrashes after a while 21:17 < Spads> and it chews up plenty of CPU too 21:17 < Spads> see if you can run some bulk tars or cpios or something 21:17 < caker> host is still making it's way through the nodes (updates every minute) 21:17 < Spads> ok 21:17 < caker> there's some slowness there, but I expected that since I haven't tuned the cpu schedulers 21:18 * Spads nods 21:18 < Spads> I'm at total lack of interactivity, btw :) 21:18 < caker> xm top says you're using 90% CPU 21:18 < Spads> yeah 21:18 < Spads> like when I launched that, I tried to type vmstat or top 21:18 < caker> out of 200% 21:18 < Spads> and couldn't even get that far 21:18 < caker> heh 21:18 < Spads> heh 21:19 < Spads> on my laptop it took a while before that happened 21:19 < caker> Spads: you're OOMing bigtime 21:19 < Spads> haha 21:19 < Spads> I expected as much 21:20 < Spads> that'll probably cut back on I/O 21:20 * caker googles for kblockd 21:20 < Spads> haha 21:20 < Spads> man, what host are we on now? 21:20 < Spads> I should watch this in lish 21:20 < caker> 56 21:20 < Spads> hehe 21:20 < Spads> well, I can pop into lish without any visible lag 21:20 < Spads> and yeah, lots of oom! 21:21 < Spads> I'm not confident that it'll ever figure out how to kill all of these 21:21 < caker> getting onto Lish in this scenario on the UML setup would be impossible 21:21 < Spads> yep 21:21 < caker> and often, people trying to do that causes sshd to freak 21:21 < Spads> hehe 21:21 < Spads> man, lish is totally responsive too 21:21 < caker> giving that "ssh_key_something or other" error on subsequent login attempts 21:21 < caker> anohter reason for the screens sessions :( 21:21 < Spads> huh 21:22 < caker> might have just been a buggie bug in rh9's sshd 21:22 < caker> even though we kept up to date with freshrpms 21:22 * Spads does a logview 21:22 < Spads> heh, it spat that out nice and fast 21:22 < caker> beautiful ... my node is 275M into swap 21:22 < caker> cycling it 21:23 < Spads> Free swap = 0kB 21:23 < Spads> Total swap = 0kB 21:23 < Spads> Free swap: 0kB 21:23 < caker> li3-242:~# killall -9 eatmem 21:23 < caker> [1]+ Killed ./eatmem 40M 10 21:23 < caker> li3-242:~# 21:23 < caker> ^-- happy again 21:23 < Spads> it's like nitrous 21:23 < Spads> it's useless while it's doing it 21:23 < caker> Total swap = 0kB ? 21:23 < Spads> but give it a breath of fresh air and BAM it's back again 21:23 < Spads> yeah, no idea 21:23 < Spads> unless you've done host-Free swap = 0kB 21:23 < Spads> Total swap = 0kB 21:23 < Spads> Free swap: 0kB 21:23 < Spads> init restarts 21:23 < Spads> damn touchpad 21:23 < caker> maybe swap device didn't mount? 21:23 * Spads shrugs 21:24 < Spads> I manually poked in swap last time we booted 21:24 < caker> er, didn't load .. the current bug I'm fighting 21:24 < Spads> after the migration 21:24 < Spads> yeah, I see that 21:24 < caker> http://lists.xensource.com/archives/html/xen-devel/2006-03/msg01332.html 21:25 < Spads> I wonder if I just leave this thing forking and OOMing 21:25 < Spads> is it still chewing up CPU? 21:25 < Spads> you know what woul dbe great actually? 21:25 < Spads> reactivate updatedb on all nodes on here 21:25 < caker> says 189% cpu now :) 21:25 < Spads> and manually poke a cron.daily 21:25 < Spads> haha 21:25 < Spads> so what if you try compiling something big? 21:25 < OvrLrd-Q> EAT MEMORAY 21:26 < caker> I'll run a while loop 21:26 < Spads> I'm just curious if you can successfully steal that CPU back from me 21:26 < OvrLrd-Q> lol and how long will that freeze up my ssh session lol :( 21:26 < mikegrb> lolz 21:26 < Spads> like, am I really eating the machine, or can anyone take it back from me? 21:26 < OvrLrd-Q> i could try 21:26 < Spads> c'mon scheduler. make us proud 21:26 < Spads> OvrLrd-Q: do a big compile, like gcc or libc or something 21:27 < OvrLrd-Q> what do you want? 21:27 < OvrLrd-Q> swap usage? cpu usage? mem usage? 21:27 < Spads> or a kernel 21:27 < Spads> any or all of the above 21:28 < Spads> wow, it's killing *lots* of bashes 21:28 < OvrLrd-Q> swap it is 21:28 < OvrLrd-Q> running 21:28 < caker> Spads: ok, we're both at 40% cpu, the rest going to the host occasionally 21:28 < Spads> sweet 21:28 < Spads> GO SCHEDULER 21:28 < caker> now we're both at 85 21:28 < caker> looks good 21:28 < Spads> Justice. 21:29 < Spads> ach 21:29 < Spads> I should do dishes now 21:29 < Spads> want me to reboot, or should we leave this running for a while? 21:29 < caker> ya, reboot 21:29 < Spads> haha ok 21:29 < Spads> c'mon lish-initiated shutdown 21:30 < Spads> do that thing 21:30 < Spads> POW 21:30 < Spads> and back up 21:30 < Spads> I hope I get swap 21:30 < OvrLrd-Q> go go perl oneliner 21:30 < Spads> haha 21:31 < OvrLrd-Q> perl -e '$_="ab"x(8*1024*1024);tr/ab/ba/ while 1' 21:31 < OvrLrd-Q> ;) 21:31 < Spads> [nick@golgatem(~)] cat /proc/swaps 21:31 < Spads> Filename Type Size Used Priority/dev/hda2 partition 262136 0 -1 21:31 < Spads> well tha's reassuring 21:32 < Spads> okay, time to clean up before Mrs. Spads gets home 22:59 -!- vodka [~knarf@ip-81-11-205-187.dsl.scarlet.be] has joined #linode-xenbeta 23:00 -!- vodka_ [~knarf@ip-81-11-192-212.dsl.scarlet.be] has quit [Ping timeout: 480 seconds] 23:20 -!- OvrLrd-Q [erwin@ip70-178-83-96.ks.ks.cox.net] has quit [Quit: ] --- Log closed Tue Mar 28 23:59:01 2006