/proc/io_status change

Sam - you'll have to consult caker or mikegrb for the "official" answer to your question, but it looks like your node has been "punished" for taking up excessive io. This is done to protect other nodes on your host box from experiencing decreased perofmance due to your node's misbehavior.

More importantly, your token_refill (number of tokens per second creditied to your total) has been cut from 512 to 50. Not only has the amount of sustained disk I/O that you can perform before hitting the stops been more-or-less halved, your recovery rate after a big I/O burst has been slashed.

Check your swap usage.

I think the downward moves are automatic. You need to raise a ticket for the limits to be raised again.

Like I said, I have little swap usage and almost always have a full io_tokens. That's why I was wondering about the situation. Having less shouldn't impact me, I was just wondering what the deal was.

I just checked my logs and the example I gave was as low as I have ever seen my io_tokens. The lowest I've seen since the change is 199931 of 20000. So I don't think I am a problem.

@Sarpy Sam:

Like I said, I have little swap usage and almost always have a full iotokens. That's why I was wondering about the situation. Having less shouldn't impact me, I was just wondering what the deal was. Now you have little swap usage and full iotokens. At some point in the recent past, your node started sucking io, which means you were into swapland most likely. Like pclissold said, you'll have to open a support ticket to get your throttle opened back up.

This is the reason my server, at idle, only uses 1/3 of it's ram :)