Leapsecondapocalypse!

I have a node running Debian 6 and was expecting to see some emails from Pingdom saying my site was down but it has been rock solid.

My Ubuntu desktop and laptop both went haywire at the same time yesterday afternoon. MySQL, Google Chrome and home Minecraft Server all started using max CPU at the same time, had to reboot both. Also checked my Linode and MySQL was fine but I rebooted it anyway.

Ubuntu 10.04 linode, no problems

$ uname -a
Linux li166-66 3.0.18-linode43 #1 SMP Mon Jan 30 11:44:09 EST 2012 i686 GNU/Linux

$ uptime
 17:15:38 up 111 days, 12:04,  1 user,  load average: 0.00, 0.01, 0.05

My Ubuntu 11.04 desktop needed rebooting 'cos VboxSVC was chewing up 110% of CPU. My CentOS 6 machines that ran qpidd. The CentOS 6 machines without qpidd and the CentOS 5 machines were all fine.

linode.pts/0% uptime
 21:36:33 up 20 days,  8:11,  1 user,  load average: 0.00, 0.01, 0.05

My bunch of random Debian 6 boxes, both linodes and vmware, were OK. Two of them were running mysql.

Had no issues here, but Hetzner reported some interesting info:

> During the night of 30.06.2012 to 01.07.2012 our internal

monitoring systems registered an increase in the level of

IT power usage by approximately one megawatt.

The reason for this huge surge is the additional switched

leap second which can lead to permanent CPU load on Linux

servers.

Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)

Completely clean across all platforms. Non-issue.

@Guspaz:

Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)

I run all the latest, home and Linode. Even on the Linode kernel 3.4.2-linode44 I saw MySQL go max

CPU, same on my two home machines a desktop and a laptop, all running Ubuntu 12.04.

The only thing I can think of that was in common is that I was running something on Java on all 3. Why would that impact MySQL? I read something about the JVM and MySQL both use a Linux thread thing called a futex. I don't program that low usually so I never used it.

@Guspaz:

Nary a problem, on any Linux box at home or Linode. Then again, I don't purposefully run ancient kernels ;)
Of course, ironically, this is a case where running an ancient kernel could have been more protective than the most current, since it could predate the introduction of the issue, which I believe was as of 2.6.22.

– David

Interesting.

In my case I did not have an issue at the exact time this was supposed to happen (midnight GMT), or 5PM PST; yet I did have my CPU on my linodes and on my local dev/test box go really high [90%-95%] at midnight between 6-30-12 and 7-1-12 for about 90 seconds. I checked the graphs for the previous 4 months and did not see that happen previously. Running Fedora 15 3.4.2-x8664-linode25 with the linodes and fedora 15 2.6.43 on the devs. MySQL version is "mysql Ver 14.14 Distrib 5.5.23, for Linux (x8664) using readline 5.1"

Yeah, both my linodes held up well. one arch one debian 6. my hetzner dedi started doing some weird things to it's gigabit connection however, two srcds instances and the minecraft server went haywire.

The Centos 6.2 box that we run cPanel websites on did not appear to suffer any problems, however the Munin graphs did show a small increase in average system load and a huge jump in committed_as memory occurring exactly at midnight. Over the next few days our external monitoring tool also showed that webpages were taking on average 50ms longer to load than the week before.

Setting the time manually from the command line immediately cleared the problem - average load and web page load times went back to the levels from the week before. (committed_as memory remains high, however).

So you may want to take a closer look at your system, even though you think everything is ok.

dovecot decided to commit suicided which required simply starting it again apart from that nothing crapped out…bugger of it was the leap second occurred at 1am where I am so pingdom work me up :(