Latest 3.0 kernel more swappy?

I see the same thing- you can see where I rebooted from latest legacy 2.6.18.8-linode22 to latest:

~~http://bryantrv.com/images/temp/swap.png" />

No other changes made, plenty of free memory near identical to yours).~~

Wow, I don't like the look of that at all.

Can an admin confirm if this is an issue to be concerned with? This kernel definitely seems to have some sort of issue.

In the meantime, according to my web server benchmarks, performance does not seem to have suffered at the current moment. Which is good :)

You can always just adjust the swappiness yourself, put a new value in /proc/sys/vm/swappiness

It's possible somebody changed the default, or the meaning of the values. Either way, trivial to change it yourself.

@nehalem:

This kernel definitely seems to have some sort of issue.
We're talking about an average of single-digit blocks per second. That's only a few kilobytes. No need for alarm here, though I'm curious about the change, too.

> We're talking about an average of single-digit blocks per second

I'd prefer an average of no blocks per second like it was for 99% of the time with the older kernel.

My last server was up for 3 months with no swapping. I'm afraid I can't be comfortable if this new kernel hammers the disk every 2 hours or so. I've seen IO usage go up into the 2000 range while it swaps and I don't like that at all. :(

FWIW, here is the Munin graph of memory for the same switch between leacy and latest :

~~http://bryantrv.com/images/temp/mem.png" />

What Munin calls inactive and active memory basically switch values.

FWIW, I do have these added lines in sysctl.conf (Debian Squeeze i386).~~

A lot has changed in Linux memory management over the past half-decade, for sure. Keep in mind that the 2.6.18 series is very very old and you will notice some (hopefully beneficial) behavioral differences in newer kernels.

Yes, more swapping seems to happen, apparently to minimize overall I/O. No, this shouldn't be a problem, and it shouldn't be thrashing more than a handful of blocks at a time.

I did a quick sampling of Linux machines I have access to (physical/virtual, desktop/server, across various providers, spanning 2.6.32 to 3.0.4), and all are at least a little bit into swap, irrespective of free memory. So, I'd say it's normal!

I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.

This is what my IO looks like for the last 24 hours:

~~http://i41.tinypic.com/2v96bns.png" />

If it stays that way tomorrow when the server is getting hammered then I'll be a very happy person.

Note: I have not made any changes to swappiness etc. I was waiting to see if it would settle down after it got used to its job of serving web pages :P

Will post back on any changes…~~

@bryantrv:

I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.
? In what way is it hitting the disk "so hard"? Edit: The Munin graph you posted earlier shows it averaging less than 1 page per second, with a peak of 44. I don't know what it means by a "page", but whatever it is, it's extremely low.

> shows it averaging less than 1 page per second

The averages are quite misleading. By looking at the averages you don't see the peaks which are in the thousands.

That aside, I got tired of the "unnecessary" swapping and adjusted the swappiness. A value of 40 gives me this:

 free -m
             total       used       free     shared    buffers     cached
Mem:           497        445         51          0         28        282
-/+ buffers/cache:        134        362
Swap:          255          0        255

That's just sweet if you ask me.

> I don't worry about it actually using swap, I just wish ti wouldn't hit the disk so hard when there is plenty of ram available. If I get anywhere near actually using all the memory, it will OOM- even when there is some still unused.

Just adjust your swappiness and forget about it :P

I have the same problem.

We have a lot of servers - while 2.6.39.1-linode34 instances run just fine without swapping, 3.0.18-linode43 instances are REALLY swappy, to the point where sometimes they just hang up completely. Like this.

~~http://cache.gyazo.com/8ad0678c01e1a29a … 59bfe2.png">http://cache.gyazo.com/8ad0678c01e1a29afdd982406b59bfe2.png" />

Notice the blank period - that's when the node was completely down.

Scary, huh? Fortunately for us, it's just one in a large cluster (that runs the same app code) so HAProxy just stop routing requests to that node, and no harm is actually done, except the slightly less overall capacity.

But it's a PITA nonetheless - in that case, we have to manually reboot from Linode Manager, cause even SSH is not responding.

I'm scared if it ever happens on our database nodes.

vm.swappiness is set to 0 on all machines, but it's not helping.~~

I set my swappiness to 20 and it was perfect. If you're still swapping maybe you genuinely need more memory.

@nehalem:

I set my swappiness to 20 and it was perfect. If you're still swapping maybe you genuinely need more memory.

That's not a logical answer.

The point is, it has been totally fine with the same amount of memory with 2.6. Plus, as you can see in the graph, memory used by app is about 60% and free + cache (= de facto free) is never less than 40% of total RAM.

What I need to know is, what makes the difference between 2.6 and 3.0, and if that's no clear, what exactly amount is "genuinely necessary memory" to determine if it would make more sense costwise to just stick with 2.6 for now.

> That's not a logical answer.

The point is, it has been totally fine with the same amount of memory with 2.6. Plus, as you can see in the graph, memory used by app is about 60% and free + cache (= de facto free) is never less than 40% of total RAM.

What I need to know is, what makes the difference between 2.6 and 3.0, and if that's no clear, what exactly amount is "genuinely necessary memory" to determine if it would make more sense costwise to just stick with 2.6 for now.

Trust me, I know your pain. That's why I started this thread in the first place. It seems that there were some changes in memory management in the new kernel (information seems scarce though). I don't know why it's like that and I hate it. I'm glad that setting my swappiness to 20 fixed the problem for me.

I'd want to see if the same app on a server with more memory behaves the same or if it stops swapping. Hence,

> maybe you genuinely need more memory

… or it will just not stop swapping.

@nehalem:

I set my swappiness to 20 and it was perfect.

For what it's worth, mine is set to 100. Unfortunately I cannot set it any higher or I would do so (heh). I barely get any swapping at all.

James

@zunzun:

For what it's worth, mine is set to 100. [] I barely get any swapping at all.[/quote]

And this is with kernel 3.x? You must be running only SSH on your server… :P

@nehalem:

And this is with kernel 3.x? You must be running only SSH on your server… :P

You might find the site useful if you need to do any curve or surface fitting: http://zunzun.com

Fitting source code is here under a BSD license: http://code.google.com/p/pyeq2/downloads/list

James