Linode.com Forum

[caker]

I'm looking for just a few Linode 64 users currently on hosts in the Fremont datacenter (host9 and up) to help test a new performance patch to the host and UML.

UML works by using ptrace, a function of Linux to intercept system calls made by applications. When your UML makes a system call, the host intercepts the call, performs the work inside UML, and null-ifies the system call on the host by converting that system call to "getpid" on the host. The problem with this is that it generates two context switches on the host.

With the [url=http://perso.wanadoo.fr/laurent.vivier/UML/
]sysemu patches[/url] on the host and in UML, it doesn't bother running "getpid", thereby reducing number of context switches from 2 to 1.

Benchmarks

Here are some "real-world" benchmarks using Mysql's run-all-tests script:
http://www.theshore.net/~caker/uml/sysemu-benchmarks.txt

In short, the tests ran 33% faster than they did without the sysemu patches.

Details

We'll migrate you to host25, keeping your slot on the previous host reserved (in case we need to migrate you back). In case you've never migrated before, it's an automated process which moves your filesystem images to the new host. No changes are required on your end, and downtime is limited to however long it takes to copy your images (usually less than 15 minutes or so for 3GB).
Set your config profile to boot the 2.4.26-linode29-1um kernel
Try to break things
If things go well for a few days, I'll consider it stable and open up the server to the public. After another few weeks, we can move the other hosts to this kernel.

Secondly, this will help test 2.6.7-rc1 on the host, an upgrade from the current 2.6.4 kernel.

Thirdly, I would also like for some of you to test the 2.4.26-linode29-1um kernel, to make sure it still functions correctly on hosts without the sysemu patches. Please let me know your experience with this kernel. On non-sysemu enabled hosts, it should perform the same as the previous linode kernels.

Let me know if you're interested...

Thanks!
-Chris

[irgeek]

Three updates in one test? You've got ball-balls Chris!

:D

-- James

[mcowger]

I'm interested....let do it!

[caker]

mcowger wrote: I'm interested....let do it!

Great. I haven't patched a 2.6-um kernel with the sysemu patches yet, so for now go with the 2.4.26-linode29-1um kernel. I'll update this thread when the 2.6 version is out.

Thanks!
-Chris

[terryr]

I'm game. What do I need to do?

Terry

[caker]

terryr wrote: I'm game. What do I need to do?

Login, shutdown, and press the migrate button. After the migration, reboot.

BTW, 5 users have moved over to host25 and so far so good.

-Chris

[terryr]

Done! Are you looking for any specific feedback?

[caker]

terryr wrote: Done! Are you looking for any specific feedback?
I pretty much know there's a performance improvement already, but hearing about that would be nice. It might be kind of difficult to pin down, unless you run specific tests and compare them against the linode28 kernel. Secondly, I wanted to generate some load on the host kernel, and the linode29 kernel, just to make sure the stability is still there.

Thanks!
-Chris

[terryr]

I loaded a fresh Fedora Core 1 distro. Noticed immediately how much faster it is. Ran locate -u which was noticeably faster. Ran top and saw something strange. About every two seconds, there's a cycle. See pics below. irq goes up to 0.9% and softirq goes up to 49.5% and iowait is 49.5%, then it all drops to 0 for 2 seconds and goes back up again for two seconds.




I don't recall this happening when I looked at top on the old host. Or did I just miss it?

Terry

[jftuga]

I am seeing stragne problems with mysql that I did not see before moving over. Whenever, I try to connect I get this message:

ERROR 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

This is because mysqld either keeps hanging or completely crashes.
It will run fine for a few hours and then just die. I can't explain why.

I will keep everyone posted.

John

[caker]

jftuga wrote: I am seeing stragne problems with mysql that I did not see before moving over. Whenever, I try to connect I get this message:

ERROR 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

This is because mysqld either keeps hanging or completely crashes.
It will run fine for a few hours and then just die. I can't explain why.
I've noticed your Linode using a lot of swap, so perhaps you're running out of memory and the OOM (Out Of Memory) killer is kicking in...

-Chris

[caker]

terryr wrote: I don't recall this happening when I looked at top on the old host. Or did I just miss it?
I'll give this a shot, too, but.. would you mind rebooting into the linode28 kernel and seeing if you can reproduce this?

I somehow doubt the SYSEMU patch would be the cause of this, or the mysql problem...

-Chris

[jftuga]

I have modified the mysql server for a small memory footprint using the my-small.cnf that comes with the src distro.

I will let you kno if this helps.

Also, I am using a lot of swap, and am considering upgrading to a 96 or 128 when they become available.

-John

[terryr]

caker wrote: I'll give this a shot, too, but.. would you mind rebooting into the linode28 kernel and seeing if you can reproduce this?

Did that yesterday with no change, but when I checked this morning, I'm at 0.1% for irq and softirq and it cycles with iowait at 0.2%. The high initial numbers didn't seem to affect performance at all.

--
Terry

[bakins]

where is the migrate button? I shutodown and clicked around but I never found it.