Linode.com Forum

[caker]

Hey guys,

I've had a few reports of a bad combination between 2.6.7 on the host (check "cat /proc/cpuinfo"), a Linode w/ Gentoo and mysql crashing often with "mysqld got signal 11".

I've also had a one or two reports of no problems what so ever, even on the same host as people that are having problems. It doesn't seem to matter if we try an older 2.4-linode kernel, nor does it matter on a 2.6-linode kernel. I've also had one report that mysql.com's binary version of mysql exhibits the same problem.

I've exhausted the things I know to try, so I'm throwing this out to you guys. What I'm looking for are reports of your Linode on a 2.6.7 host, Gentoo, and mysql that behaves. And more importantly, mysql's that don't behave with backtraces/gdb output, or any other relevant information.

Thanks,
-Chris

[caker]

Few more tidbits of info:

Code: localhost sql-bench # epm -qa|grep glibc
glibc-2.3.3.20040420
localhost sql-bench # /usr/sbin/mysqld -V
/usr/sbin/mysqld  Ver 4.0.20-debug for pc-linux-gnu on i686 (Gentoo Linux mysql-4.0.20)
Running the ./test-connect script, I'm able to get mysqld to segfault:

Code: Program received signal SIGSEGV, Segmentation fault.
0x40382521 in select () from /lib/libc.so.6
(gdb) backtrace
#0  0x40382521 in select () from /lib/libc.so.6
#1  0x00000005 in ?? ()
#2  0xbffffaa0 in ?? ()
#3  0xbffffaa0 in ?? ()

So it looks like it could be a libc issue. Still digging...

[caker]

Ok, I retried with Mysql.com's "mysql-standard-4.0.20-pc-linux-i686" version with the "test-connect" tests and guess what? It completed the tests just fine.

I'll retest a few more times...

[darfsnuzal]

Hey everyone,

Well, I hate to bring up bad news but this exact situation has been occuring for me since I started using Linode about a month and a half ago. I've been pulling what's left of my hair out over this. I've googled and googled, recompiled and tested to no avail.

What's happening is that mysqld is hanging at random times. I'm running gentoo but not using 2.6.7:

Code:
Linux cartman 2.4.26-linode31-1um #1 Thu Jul 8 18:24:02 EDT 2004 i686 UML User Mode Linux GNU/Linux


I have been able to re-create this by running the connection test in the mysql package. Mysqld throws a segfault and locks up hard. It doesn't do this right away all the time. Sometimes it will finish 50% of the test, sometimes 70% sometimes 10%. Here is the stack trace that I'm getting:

0x80fded1 handle_segfault + 641
0x40028685 _end + 936406917
0x810a771 handle_one_connection + 33
0x40023041 _end + 936384833
0x40269b8a _end + 938771594

I'm running:
Code:
mysqld  Ver 4.0.20-debug for pc-linux-gnu on i686 (Gentoo Linux mysql-4.0.20-r1

Also:
Code:
cartman mysql # epm -qa|grep glibc
glibc-2.3.3.20040420-r1

During normal runs, this DB is maybe getting 10 connections an hour so recreating the segfault can take forever if I don't run the test scripts.

Does anyone have any ideas on how to correct this? Please? Help? I've tried so many things I don't know where to begin describing them...

[caker]

Two things: it's 2.6.7 on the host, regardless of which Linode kernel version you run.

Also, can you please try the mysql.com's mysql package? That seems to have fixed it when I was struggling with this. Can you verify?

Thanks,
-Chris

[darfsnuzal]

Chris,

Will do. Thanks for the response!

[darfsnuzal]

Chris,

I tried the mysql binary package like you asked. With the statically linked binary, I don't see all out crashes like I did before.

One very interesting note. During the testing over the last several days, I've noticed that after working with mysqld, that *sometimes* within a few minutes of starting or stoping mysqld I will get a segfault from just about any command I type at the command line. For example: I just retested the gentoo source release of mysqld and this happened:

Code:
cartman mysql # ll
total 40524
drwxr-xr-x  2 mysql mysql     1024 Aug 31 16:16 .
drwxr-xr-x  6 root  root      3072 Aug 31 03:10 ..
-rw-rw----  1 mysql mysql    41175 Aug 31 19:05 mysql.err
-rw-rw----  1 mysql mysql 37650787 Aug 31 18:42 mysql.log
-rw-r--r--  1 root  root         0 Aug  8 20:59 mysql.out
-rw-rw----  1 mysql mysql  3272789 Aug 31 18:43 mysqld.err
-rw-r--r--  1 root  root        53 Aug 31 15:41 mysqld.stack.1
-rw-r--r--  1 root  root        53 Aug 31 16:15 mysqld.stack.2
-rw-r--r--  1 root  root    354822 Aug 31 15:40 mysqld.sym
-rw-r--r--  1 root  root        53 Aug 22 16:27 stack
cartman mysql # tail -f mysql.err
040831 19:04:37  InnoDB: Starting shutdown...
040831 19:04:39  InnoDB: Shutdown completed
040831 19:04:39  /usr/local/mysql/bin/mysqld: Shutdown Complete

040831 19:04:39  mysqld ended

040831 19:05:07  mysqld started
/usr/local/mysql/bin/mysqld: ERROR: unknown variable 'defaults-file=/etc/mysql/my.cnf'
040831 19:05:07  mysqld ended

040831 19:07:15  mysqld started
Warning: Ignoring user change to 'mysql' because the user was set to 'mysql' earlier on the command line

cartman mysql # ll
Segmentation fault
cartman mysql #


ll is just an alias for 'ls -la'. I have had this happen when I've used tail, ls and cd. It *never* happens except when I'm trying to get mysqld to behave. It's occurred at least five times since yesterday and I vaugely remember it happening about a month ago when this problem was starting to go from bad to just plain silly.

Ideas?

[chuyeow]

OK I seem to have solved my problem with MySQL crashing.

I didn't go about rigorously following the stack trace. Rather, I discovered that mysql was constantly forking which was kinda weird. I figured that too many requests were coming in and mysqld just crashed when it couldn't handle it. Possibly also because I'm on a Linode 64 plan so resources are limited.

So I went to tweak my mysqld's server variables. The most important change (to my.cnf) was to turn on the thread cache (it is 0 by default).
Code:
set-variable    = thread_cache_size=40


But first, you have to see whether mysql is creating more thread than it can handle. I logged into the mysql server and did a SHOW STATUS. Turned out that the no. of Threads_created was constantly increasing (run SHOW STATUS a few times). Mine was increasing all the time so I turned on the thread cache using SET GLOBAL thread_cache_size=40; and immediately the Threads_created stopped increasing. A restart of mysqld later, the thread count is very low (5-10, right now it's 7) compared to figures like 380 before. And mysqld hasn't crashed a single time since this tweak.


I don't think this is the root cause of the problem though, but part of the reason, so try tweaking your mysql settings and see if it helps at all, at least until the reason for the bug is found you have a working mysqld!

[darfsnuzal]

Amazing. Simply amazing. That did the trick. I ran the sql-bench tests twice without problems. I tweaked ten other variables but not that one. Thank you very much chuyeow!

[chuyeow]

No problem! Just glad I could help. I agonized over this one for half a day. a night, and the next morning, believe it or not. Good that it saved someone else the troubles.

I have to ask, did u do the SHOW STATUS thing to see how many threads were created and whether it was increasing?

[caker]

chuyeow wrote: OK I seem to have solved my problem with MySQL crashing.
Just to verify, you're running Gentoo?

darfsnuzal wrote: That did the trick. I ran the sql-bench tests twice without problems.
And this was with Gentoo's (not mysql.com's) package?

Thanks!
-Chris

[chuyeow]

Yup I am running Gentoo (with the latest 2.4 kernel). (Gentoo's mysql package, just in case.)

[darfsnuzal]

Yep, that was using gentoo's mysl. In fact to make sure, I tested before adding the variable to /etc/mysql/my.cnf and it crashed hard. I added the variable and ran the tests twice again and they didn't fail.

For the record, I did observe the rapidly increasing thread count phenomena that chuyeow described. Now the counts say very low; between 4-10 threads.

[marcus]

Bingo!

I can't believe how lucky I was to stumble upon this thread - I have been having problems with a stray java process going wild and sucking 100% CPU as it was trying to repeatedly access a SQL connection, which mysql wasn't prepared to do after exhibiting this behavior.

Before adding the variable mentioned above I could not successfully run the test-connect script.

After adding the variable to my.cnf I can run the test-connect script A-OK.

I was previously using mysql 4.0.20, then 4.0.22, then am currently using 4.1.8 (I tried a lot before I found this post..)

Thanks guys!

[marcus]

marcus wrote: Bingo!

I can't believe how lucky I was to stumble upon this thread - I have been having problems with a stray java process going wild and sucking 100% CPU as it was trying to repeatedly access a SQL connection, which mysql wasn't prepared to do after exhibiting this behavior.

Before adding the variable mentioned above I could not successfully run the test-connect script.

After adding the variable to my.cnf I can run the test-connect script A-OK.

I was previously using mysql 4.0.20, then 4.0.22, then am currently using 4.1.8 (I tried a lot before I found this post..)

Thanks guys!

uh-oh.. i'm worried now I may have spoken too soon: although the test-connect script did complete once successfully, I have encountered the same error I have been ever since I migrated linodes (and got a new kernel version?).

Upon running the test-connect script again, it failed midway through.

Again, this is using Gentoo's 4.1.8 version of mysql. It seems like people have been having good luck with 4.0.22, so maybe I'll try to (gasp) downgrade to that? Are there any other configuration switches I can try putting in my.cnf? Any other misc. ideas?