Disk Image Size Causing MySQL Crash?
I've just noticed my disk storage is at 100% capacity with 256MB used up by Swap and the remaining used by my Ubuntu disk image. Could lowering the amount of memory available to my Ubuntu disk be helpful in preventing these crashes by freeing up RAM?
9 Replies
df
Have you checked the MySQL and system logs to see if there are any errors in there indicating why the db server has crashed? Hard disk space and RAM are mutually exclusive (unless you're using a RAM disk or something non-Linode default).
There is one error that is occasionally logged during restart regarding. It reports my server not being able allocate enough memory to InnoDB buffer pool, which is only set to the default 128.0M
About the quote below:
@The Other Air Force:
The storage allocated and the storage free in the Linode manager only indicates how much you have allocated for the Linode to potentially use.
Does this mean I should continue allocating 100% of my disk space to make full use of my Linode?
Back to MySQL what are the results of:
> free -m
Is it actually the MySQL server that is having problems or is it the application that is trying to connect with it? Can you tell us exactly what error messages you are getting or what is leading you to believe that there is a problem with the server vs. the application?
https://tools.percona.com/wizard
The application itself isn't very memory efficient. It's a Wordpress theme that uses some inefficient queries to access a fairly large database. But the site wasn't crashing with nearly as much frequency on my previous Linode, which was much smaller and running only a 32 bit instance of MySQL. This suggests to me that the problem is in my server configs.
Below is what free -m returns. Note I've moved 100MB to Swap since my original post.
total used free shared buffers cached
Mem: 1992 917 1075 26 10 200
-/+ buffers/cache: 706 1285
Swap: 351 0 351
I should also note, DB crashes were exacerbated even more when I switched all my DB tables from MyISAM to InnoDB two days ago. I've since switched back to MyISAM but the DB is still crashing with the same regularity.
Feb 10 15:27:13 mylinode kernel: apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
The previous entry was logged 2 minutes prior
Feb 10 15:25:01 mylinode CRON[5655]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
I don't think this is the issue though, because if I'm reading the log file correctly, it looks like I have an absurd amount of Apache2 processes running. Monitoring my server using 'top', I typically only see 1-5 Apache processes running at a time.
Feb 10 15:27:13 mylinode kernel: 16126 pages reserved
Feb 10 15:27:13 mylinode kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Feb 10 15:27:13 mylinode kernel: [ 1456] 0 1456 5071 1 14 273 0 upstart-udev-br
Feb 10 15:27:13 mylinode kernel: [ 1461] 0 1461 12325 1 27 145 -1000 systemd-udevd
Feb 10 15:27:13 mylinode kernel: [ 2444] 0 2444 2560 0 6 573 0 dhclient
Feb 10 15:27:13 mylinode kernel: [ 2649] 102 2649 9781 1 23 88 0 dbus-daemon
Feb 10 15:27:13 mylinode kernel: [ 2727] 0 2727 10865 13 27 69 0 systemd-logind
Feb 10 15:27:13 mylinode kernel: [ 2736] 0 2736 15343 28 32 142 -1000 sshd
Feb 10 15:27:13 mylinode kernel: [ 2738] 0 2738 4787 0 13 43 0 atd
Feb 10 15:27:13 mylinode kernel: [ 2739] 0 2739 5916 23 16 43 0 cron
Feb 10 15:27:13 mylinode kernel: [ 2753] 101 2753 63963 77 27 89 0 rsyslogd
Feb 10 15:27:13 mylinode kernel: [ 2817] 0 2817 3920 1 13 146 0 upstart-file-br
Feb 10 15:27:13 mylinode kernel: [ 2819] 0 2819 4015 0 13 263 0 upstart-socket-
Feb 10 15:27:13 mylinode kernel: [ 2821] 105 2821 472444 47076 358 87305 0 mysqld
Feb 10 15:27:13 mylinode kernel: [ 2867] 0 2867 24659 53 47 390 0 sendmail-mta
Feb 10 15:27:13 mylinode kernel: [ 3158] 0 3158 78253 2750 90 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 3187] 0 3187 66396 1780 31 0 0 fail2ban-server
Feb 10 15:27:13 mylinode kernel: [ 3229] 0 3229 3199 40 11 0 0 getty
Feb 10 15:27:13 mylinode kernel: [ 3345] 103 3345 7863 150 20 0 0 ntpd
Feb 10 15:27:13 mylinode kernel: [ 3368] 0 3368 26446 273 53 0 0 sshd
Feb 10 15:27:13 mylinode kernel: [ 3404] 1000 3404 26446 271 52 0 0 sshd
Feb 10 15:27:13 mylinode kernel: [ 3405] 1000 3405 5563 420 16 0 0 bash
Feb 10 15:27:13 mylinode kernel: [ 4307] 33 4307 80672 12282 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4343] 33 4343 116884 48398 180 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4433] 33 4433 84282 16034 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4440] 33 4440 83661 16264 116 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4457] 33 4457 81064 12773 110 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4458] 33 4458 83032 14720 114 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4466] 33 4466 87206 18817 121 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4503] 33 4503 113967 46300 178 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4505] 33 4505 85272 16833 118 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4512] 33 4512 85312 17147 120 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4565] 33 4565 95000 26641 136 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4566] 33 4566 84041 15744 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4626] 33 4626 84444 16130 117 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 4732] 33 4732 95024 27384 139 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5056] 33 5056 79887 11825 108 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5283] 33 5283 81158 12768 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5284] 33 5284 79773 8968 103 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5285] 33 5285 81363 10349 106 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5286] 33 5286 81152 12778 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5330] 33 5330 81157 12702 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5331] 33 5331 82528 14922 114 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5372] 33 5372 79795 9117 103 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5373] 33 5373 81144 12785 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5374] 33 5374 81160 12785 109 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5674] 33 5674 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5676] 33 5676 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5677] 33 5677 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5680] 33 5680 79744 8865 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5681] 33 5681 79746 8868 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5682] 33 5682 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5683] 33 5683 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5688] 33 5688 79741 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5689] 33 5689 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5690] 33 5690 79742 8863 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5691] 33 5691 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5692] 33 5692 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5694] 33 5694 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5698] 33 5698 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5700] 33 5700 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5704] 33 5704 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5705] 33 5705 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5706] 33 5706 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5707] 33 5707 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5708] 33 5708 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5711] 33 5711 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5713] 33 5713 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5715] 33 5715 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5718] 33 5718 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5719] 33 5719 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5722] 33 5722 79806 8864 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5725] 33 5725 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5727] 33 5727 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5729] 33 5729 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5731] 33 5731 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5733] 33 5733 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5736] 33 5736 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5737] 33 5737 79745 8869 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5738] 33 5738 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5739] 33 5739 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5740] 33 5740 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5743] 33 5743 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5747] 33 5747 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5749] 33 5749 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5750] 33 5750 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5751] 33 5751 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5754] 33 5754 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5757] 33 5757 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5759] 33 5759 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5761] 33 5761 79806 8849 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5762] 33 5762 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5763] 33 5763 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5766] 33 5766 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5769] 33 5769 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5771] 33 5771 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5772] 33 5772 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5775] 33 5775 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5777] 33 5777 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5779] 33 5779 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5780] 33 5780 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5781] 33 5781 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5784] 33 5784 79742 8855 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5787] 33 5787 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5788] 33 5788 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5790] 33 5790 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5793] 33 5793 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5794] 33 5794 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5796] 33 5796 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5800] 33 5800 79744 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5801] 33 5801 79743 8863 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5802] 33 5802 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5803] 33 5803 79746 8868 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5804] 33 5804 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5806] 33 5806 79743 8851 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5808] 33 5808 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5811] 33 5811 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5815] 33 5815 79744 8867 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5816] 33 5816 79743 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5819] 33 5819 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5820] 33 5820 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5822] 33 5822 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5825] 33 5825 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5826] 33 5826 79742 8849 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5828] 33 5828 79744 8850 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5831] 33 5831 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5833] 33 5833 79743 8866 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5836] 33 5836 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5837] 33 5837 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5839] 33 5839 79806 8864 102 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5841] 33 5841 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5842] 33 5842 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5843] 33 5843 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5847] 33 5847 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5848] 33 5848 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5850] 33 5850 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5851] 33 5851 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5855] 33 5855 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5857] 33 5857 79741 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5858] 33 5858 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5859] 33 5859 79742 8864 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5864] 33 5864 79600 8081 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5865] 33 5865 79742 8495 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5866] 33 5866 79742 8556 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5868] 33 5868 79598 8411 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5869] 33 5869 79672 8309 101 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5873] 33 5873 79600 8370 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5874] 33 5874 79600 8257 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5877] 33 5877 79272 7190 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5878] 33 5878 79272 7122 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5879] 33 5879 79340 7401 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5881] 33 5881 79272 6968 99 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5885] 33 5885 79600 8260 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5887] 33 5887 79340 7175 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5888] 33 5888 79055 6328 98 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5891] 33 5891 78957 4745 95 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5892] 33 5892 78794 3688 90 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5894] 33 5894 79600 8433 100 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5896] 33 5896 79022 5810 97 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5898] 33 5898 78598 2088 88 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5899] 33 5899 78308 1476 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5901] 33 5901 78498 1910 87 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5902] 33 5902 78533 2140 87 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5905] 33 5905 78308 1538 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5908] 33 5908 78308 1470 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5909] 33 5909 78308 1470 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5910] 33 5910 78267 1453 81 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5912] 0 5912 78255 1444 78 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5913] 33 5913 78259 1446 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5914] 33 5914 78259 1446 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5915] 33 5915 78271 1462 84 0 0 apache2
Feb 10 15:27:13 mylinode kernel: [ 5916] 33 5916 78265 1451 80 0 0 apache2
Feb 10 15:27:13 mylinode kernel: Out of memory: Kill process 2821 (mysqld) score 224 or sacrifice child
I have also taken your suggestion and turned keepAlive off. Though does this now mean that every CSS file and every image request will now need to spawn a new Apache thread?
Also, about my previous syslog excerpt. My previous MaxKeepAliveTimeout was 5 sec which is fairly generous, so does the number of Apache processes typically suggest bot activity or a DNS type of attack? For my site to get that many new HTTP requests simultaneously doesn't seem normal.
Thanks obs and Other Air Force for your time so far!
When someone requests a resource from your site keep alives keep the connection open for x number of seconds so that subsequent requests can use the same connection, however this means the process can't serve someone else. With keep alives off each resource uses a new connection. The best settings will depend on what you're serving, if your pages have a lot of files then keep alives can be useful if you're serving downloads then there's not much point.
The number of processes just means you had a traffic burst, could simply be google crawling your site, you'd have to check your access logs.
I created a pull request to update the docs.