Had MySQL killed b/c of low memory. PHP5-fpm uses 900+ MB.

Hello,

I logged into my linode yesterday and realized MySQL wasn't on. It had been killed (from reading dmesg) because it was out of memory. My Swap was also 100%. I believe it was PHP fpm taking up all of the memory. I changed the configs and am curious if I am missing anything else to help performance so this won't happen again, especially at times when more traffic hits. Even with this config, Longview shows php5-fpm using 900MB-1GB of memory and I only have a 1 GB linode.

Current ubuntu status when i just logged in shows memory usage: 55% and swap usage: 14%, 155 processes.

I am currently doing a FPM pool for each subdomain and domain hosted (there's 2 main domains hosted). Is that best or should I actually just use 1?

Any help is appreciated! Sanitized configs below.

Nginx

user www-data;
worker_processes 8;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

FPM pool 1 (main site)

[domain.com]
listen = /var/run/php5-fpm/domain.com.socket
listen.backlog = -1

; Unix user/group of processes
user = myaccount
group = www-data

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 4
pm.process_idle_timeout = 10s
pm.min_spare_servers = 4
pm.max_spare_servers = 10
pm.status_path = /status
pm.max_requests = 500
security.limit_extensions = .php .html .run
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

PHP fpm pool 2 (admin, low usage subdomain)

[admin.domain.com]

listen = /var/run/php5-fpm/admin.domain.com.socket
listen.backlog = -1

; Unix user/group of processes
user = myaccount
group = www-data

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 20
pm.max_requests = 500

; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

fpm pool 3 (admin subdomain, little more usage, still for public use)

[races.domain.com]
listen = /var/run/php5-fpm/races.domain.com.socket
listen.backlog = -1

; Unix user/group of processes
user = myaccount
group = www-data

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 10
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.max_requests = 200
security.limit_extensions = .php .html .run
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

fpm pool domain 2, public use. wordpress

[domain2.com]
listen = /var/run/php5-fpm/domain2.com.socket
listen.backlog = -1

; Unix user/group of processes
user = myaccount
group = www-data

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 25
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 10
pm.process_idle_timeout = 10s
pm.max_requests = 500

; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

fpm pool staging domain

[test.domain.com]
listen = /var/run/php5-fpm/test.domain.com.socket
listen.backlog = -1

; Unix user/group of processes
user = myaccount
group = www-data

; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 5
pm.max_requests = 100
security.limit_extensions = .php
; Pass environment variables
env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

MySQL my.cnf

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1
#
# * Fine Tuning
#
key_buffer              = 16M
max_allowed_packet      = 1M
thread_stack            = 64K
thread_cache_size       = 8
sort_buffer = 64K
net_buffer_length = 2K
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
table_cache            = 4

query_cache_limit       = 1M
query_cache_size        = 16M

log_error = /var/log/mysql/error.log

expire_logs_days        = 10
max_binlog_size         = 100M

[mysqldump]
quick
quote-names
max_allowed_packet      = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer              = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

7 Replies

I don't use php-fpm because most of the people who recommend php-fpm do so based on a faulty understanding of how to measure Apache memory usage, which isn't to say that there aren't good reasons to use it, I just haven't seen any that apply to me.

That said, running multiple pools seems like a mixed bag. Each pool is going to take X MB just to run, and then additional memory for the application code and extensions they load. With multiple pools, each pool might use less memory than they would if everything ran in the same pool, but that's going to be offset or overshadowed by the fact that you have more of them.

The bigger issue though, is that multiple pools make resource management more difficult. With a single pool, you can set a resource cap that limits the total amount of RAM taken up by all the php apps on the system and you can be confident that whatever app(s) are experiencing the highest load at any given time will be able to get as much RAM as possible without taking too much. With multiple pools, you have to make a static allocation of resources among pools. If you do that in a way that guarantees that PHP can never take up too much RAM, then you are also guaranteeing that in some situations, RAM resources will sit idle even when they are needed.

If I were you, I'd probably cut this down to one or two pools (the second for your admin stuff), and set things up so not more than ~24 children were allowed in the main pool, and maybe 4 or so in the admin pool. My thinking is that when I've done my own testing, there doesn't seem to be any point to having more than active 3-4 workers per CPU core (if even that). With that, I see ~100% usage of all cores under a load test. Once CPU utilization is maxed out, having more workers can actually work against you because the extra workers end up taking up more RAM, pushing stuff out of disk cache, which then requires more disk I/O, which slows things down, by requiring the CPUs to wait more for disk.

I merged the pools (so only 2) and adjusted the children and yes, memory usage is now hovering (at least for 2 days) at 35-45%. Thanks! Everything's good for now…guess I'll see what happens when I get another spike in traffic. I also am worried what the memory usage is going to look like once I start deploying Ruby apps :/

xcrunner529,

If you're already maxing things out now you should probably up-size your Linode to 2GB before your regular traffic starts going up, and before you deploy more sites or resources.

MSJ

Are you kidding? Of course a Linode that I control and have full resources on is better. It's just a matter of optimizing the right things and picking the right config. With Shared Hosts we ran into execution limits, usually couldn't backup stuff because of those limits, page publishing from the CMS would get killed, I was stuck with only PHP/MySQL effectively, e-mail was unreliable (and with GoDaddy I couldn't use any external mail services), overpriced SSL certs + private IP charges, no real ability to fix slowness from oversubscribed servers, etc.

A box we control allows me to use more modern platforms, do more server-side caching, run a more efficient web server, etc.

I suppose, but this was all running on shared hosting before…mostly fine but a little slow sometimes. I finally convinced them to move to a Linode so I could start developing better, more modern apps and have more control over things such as mailing and the resources we get. Having to move to a bigger Linode just to make what worked before work doesn't seem right and I wouldn't be able to convince them to pay more already. If all else fails, I'll just go back to straight Apache and php like GoDaddy had if this isn't going to be better and faster. I thought it would be.

@xcrunner529:

I finally convinced them to move to a Linode so I could start developing better, more modern apps and have more control over things such as mailing and the resources we get
And you did that with no testing, no first hand knowledge that it will work better, or anything?

You should only really have multiple pools if you have a different php configuration per pool (Talking php.ini here, not php-fpm.conf) OR you're doing some heavy caching with one app that another one can't use type thing, otherwise it's a wasted overhead. Could have workers sitting in one pool doing nothing while another pool needs more.

I run ~20 different php websites off one web front end, I have one pool.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct