Maxed out Swap

performance

Hi all,

I've got an odd problem with a 2GB linode running Debian 7.5

The server is used as a basic web/mail/mysql/ftp machine hosting a few low traffic websites.

I've noticed that the machine is using 100% Swap in longview and 74% memory whilst similar machines are using 0% Swap and 24% memory.

Can anyone give me some pointers on where to look to find out what is using the memory and causing swap to run at 100%?

Thanks.

6 Replies

forum:mattltm 9 years ago

I should add that even after a reboot, the swap starts at 60% and climbs back up to 100% within a few hours.

forum:obs 9 years ago

What's the output ofps aux --sort '-rss'

forum:mattltm 9 years ago

````
root@viking:/# ps aux --sort '-rss'
STAT START TIME COMMAND
Ssl 12:19 0:09 /usr/sbin/clamd -c /etc/clamav/clamd.conf
S 12:19 0:00 php-fpm: pool web9
S 12:19 0:00 php-fpm: pool web16
Sl 12:20 0:13 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/my
S 12:19 0:00 php-fpm: pool web18
S 12:19 0:06 php-fpm: pool web8
S 12:19 0:06 php-fpm: pool web8
S 12:19 0:00 php-fpm: pool web9
S 12:19 0:04 php-fpm: pool web7
S 12:19 0:00 php-fpm: pool web2
S 12:19 0:04 php-fpm: pool web7
S 13:45 0:01 /usr/sbin/amavisd-new (ch7-avail)
S 13:30 0:01 /usr/sbin/amavisd-new (ch10-avail)
S 12:19 0:00 php-fpm: pool web2
S 13:31 0:00 /usr/sbin/amavisd-new (ch9-avail)
S 13:40 0:00 /usr/sbin/amavisd-new (ch9-avail)
Ss 12:20 0:02 /usr/sbin/spamd --create-prefs --max-children 5 --helper-home-dir -d --pidfile=/v
S 12:20 0:00 spamd child
S 12:20 0:00 spamd child
S 13:13 0:01 php-fpm: pool web7
S 12:19 0:00 php-fpm: pool web3
S 12:19 0:01 php-fpm: pool web22
S 12:19 0:00 php-fpm: pool web16
S 12:19 0:00 php-fpm: pool web11
S 13:42 0:00 /usr/sbin/amavisd-new (ch7-avail)
S 12:19 0:01 php-fpm: pool web22
S 12:19 0:00 php-fpm: pool web4
S 12:19 0:05 python -c from pyovpn.sagent.sagententry import openvpnas ; openvpnas() --logfil Ss 12:19 0:01 /usr/sbin/amavisd-new (master) Ss 12:19 0:00 /usr/sbin/apache2 -k start S 12:19 0:00 php-fpm: pool web11 S 12:19 0:00 php-fpm: pool web4 S 12:19 0:00 php-fpm: pool web19 S 12:20 0:02 python -c from pyovpn.cserv.wserventry import start ; start() -no -u openvpnas S 14:15 0:00 /usr/bin/php-cgi -d openbasedir=/var/www/clients/client22/web21/web:/var/www/cli
S 12:19 0:00 php-fpm: pool web18
S 12:19 0:00 php-fpm: pool web19
Ss 12:24 0:00 /usr/bin/php-cgi -d magicquotesgpc=off -d session.savepath=/usr/local/ispconfi Ss 12:19 0:04 linode-longview S 12:20 0:00 python -c from pyovpn.log.logworker import start ; start() S 13:32 0:00 /usr/sbin/apache2 -k start S 13:38 0:00 /usr/sbin/apache2 -k start S 13:37 0:00 /usr/sbin/apache2 -k start S 13:56 0:00 /usr/sbin/apache2 -k start S 13:13 0:00 /usr/sbin/apache2 -k start S 13:13 0:00 /usr/sbin/apache2 -k start S 14:07 0:00 /usr/sbin/apache2 -k start S 12:40 0:00 /usr/sbin/apache2 -k start S 12:41 0:00 /usr/sbin/apache2 -k start S 12:21 0:00 /usr/sbin/apache2 -k start S 12:20 0:02 python -c from pyovpn.sagent.iptworker import start ; start() S 12:19 0:00 php-fpm: pool web5 S 12:20 0:00 python -c from pyovpn.sagent.iptworker import start6 ; start6() S 12:24 0:00 /usr/bin/php-cgi -d magicquotesgpc=off -d session.savepath=/usr/local/ispconfi
S 12:19 0:01 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
Ss 12:19 0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
S 12:19 0:00 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
Ss 12:19 0:00 munin-node
Sl 12:19 0:09 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock
Ss 12:19 0:00 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -s -q start
S 13:45 0:00 smtpd -n smtp -t inet -u -c -o stress= -s 2
Ssl 12:19 0:00 /usr/sbin/named -u bind
S 12:19 0:00 /usr/sbin/apache2 -k start
S 12:19 0:00 /usr/sbin/fcgi-pm -k start
S 14:11 0:00 smtpd -n 127.0.0.1:10025 -t inet -u -c -o contentfilter= -o localrecipientmaps S 14:00 0:00 trivial-rewrite -n rewrite -t unix -u -c S 14:10 0:00 dovecot/imap Ss 12:19 0:04 /usr/bin/freshclam -d --quiet --config-file=/etc/clamav/freshclam.conf S 14:00 0:00 proxymap -t unix -u S 14:09 0:00 dovecot/imap S 14:09 0:00 dovecot/imap S 12:20 0:00 tlsmgr -l -t unix -u -c Ss 14:15 0:00 sshd: pengimo [priv] S 14:09 0:00 dovecot/imap Ss 14:16 0:00 -bash S 12:34 0:00 dovecot/imap S 13:42 0:00 dovecot/imap S 14:11 0:00 dovecot/imap S 12:19 0:00 vlogger (access log) S 12:20 0:00 qmgr -l -t fifo -u S 14:15 0:00 dovecot/imap S 12:49 0:00 php-fpm: pool web16 S 14:09 0:00 dovecot/imap S 13:50 0:00 dovecot/imap S 13:49 0:00 dovecot/imap S 14:11 0:00 dovecot/imap S 12:20 0:00 php-fpm: pool web19 S 14:09 0:00 dovecot/imap S 13:47 0:00 dovecot/imap S 13:49 0:00 dovecot/imap S 12:19 0:00 php-fpm: pool www S 12:19 0:00 php-fpm: pool www S 12:19 0:00 php-fpm: pool web13 S 12:19 0:00 php-fpm: pool web5 S 12:19 0:00 php-fpm: pool web1 S 12:19 0:00 php-fpm: pool web12 S 12:19 0:00 php-fpm: pool web6 S 12:19 0:00 php-fpm: pool web6 S 12:19 0:00 php-fpm: pool web23 S 12:19 0:00 php-fpm: pool web23 S 12:19 0:00 php-fpm: pool web1 S 12:19 0:00 php-fpm: pool web13 S 12:19 0:00 php-fpm: pool web12 S 12:19 0:00 php-fpm: pool web3 S 13:42 0:00 dovecot/imap S 13:50 0:00 dovecot/imap-login S 13:50 0:00 dovecot/imap-login S 13:50 0:00 dovecot/imap S 13:46 0:00 dovecot/imap S 13:44 0:00 dovecot/imap S 14:09 0:00 dovecot/imap S 13:54 0:00 dovecot/imap S 14:15 0:00 dovecot/auth -w S 14:07 0:00 dovecot/imap S 14:09 0:00 dovecot/imap-login S 14:09 0:00 dovecot/imap-login S 13:46 0:00 dovecot/imap-login S 13:47 0:00 dovecot/imap-login S 14:09 0:00 dovecot/imap-login S 14:09 0:00 dovecot/imap-login S 14:11 0:00 dovecot/imap-login S 13:49 0:00 dovecot/imap-login S 14:11 0:00 dovecot/imap-login Sl 12:20 0:01 /usr/bin/monit -c /etc/monit/monitrc S 14:07 0:00 dovecot/imap-login S 13:44 0:00 dovecot/imap-login S 12:26 0:00 dovecot/imap S 14:02 0:00 dovecot/imap S 13:49 0:00 dovecot/imap-login S 14:02 0:00 dovecot/imap-login S 14:15 0:00 dovecot/imap-login S 13:52 0:00 dovecot/imap-login S 12:30 0:00 dovecot/imap S 12:21 0:00 dovecot/imap S 12:26 0:00 dovecot/imap-login S 12:20 0:00 dovecot/auth Ss 12:20 0:00 pure-ftpd (SERVER) Ss 12:20 0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:103 S 12:20 0:00 dovecot/config S 14:00 0:00 pickup -l -t fifo -u -c S 14:15 0:00 showq -t unix -u -c S 12:20 0:00 anvil -l -t unix -u -c Ss 12:20 0:00 /usr/lib/postfix/master Ssl 12:20 0:01 /usr/local/bin/znc -d /var/lib/znc S 14:16 0:00 sshd: pengimo@pts/0 S 14:16 0:00 bash S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin S 14:16 0:00 sudo su Ss 12:20 0:00 /usr/sbin/sshd S 14:16 0:00 su S 12:19 0:00 /bin/sh /usr/bin/mysqldsafe
S 12:19 0:00 /usr/lib/gamin/gamserver R+ 14:16 0:00 ps aux --sort -rss Sl 12:19 0:00 /usr/bin/memcached -m 64 -p 11211 -u nobody -l 127.0.0.1 Ss 12:19 0:00 /usr/sbin/cron Ss 12:20 0:00 /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf Sl 12:19 0:00 /usr/sbin/rsyslogd -c5 S 12:20 0:00 dovecot/log Ss 12:19 0:00 udevd --daemon S 12:20 0:00 dovecot/anvil Ss 12:19 0:00 /sbin/rpc.statd Ss+ 12:20 0:00 /sbin/agetty hvc0 S 12:20 0:00 iptables-restore -n S 14:15 0:00 dovecot/ssl-params Ss 12:19 0:00 /sbin/rpcbind -w S 12:19 0:00 udevd --daemon S 12:19 0:00 udevd --daemon S 12:20 0:00 logger -t mysqld -p daemon.error Ss 12:19 0:00 startpar -f -- jailkit Ss 12:19 0:00 init [2] S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin S 12:20 0:00 openvpn-openssl --errors-to-stderr --config stdin Ss 12:19 0:00 /usr/bin/dbus-daemon --system Ssl 12:19 0:00 /usr/sbin/jksocketd
S 12:19 0:00 [kthreadd]
S 12:19 0:00 [ksoftirqd/0]
S 12:19 0:00 [kworker/0:0]
S< 12:19 0:00 [kworker/0:0H]
S 12:19 0:00 [kworker/u4:0]
S 12:19 0:02 [rcusched] S 12:19 0:00 [rcubh]
S 12:19 0:00 [migration/0]
S 12:19 0:00 [migration/1]
S 12:19 0:00 [ksoftirqd/1]
S 12:19 0:00 [kworker/1:0]
S< 12:19 0:00 [kworker/1:0H]
S< 12:19 0:00 [khelper]
S 12:19 0:00 [kdevtmpfs]
S< 12:19 0:00 [netns]
S< 12:19 0:00 [perf]
S 12:19 0:00 [xenwatch]
S 12:19 0:00 [xenbus]
S< 12:19 0:00 [writeback]
S< 12:19 0:00 [crypto]
S< 12:19 0:00 [kintegrityd]
S< 12:19 0:00 [bioset]
S< 12:19 0:00 [kblockd]
S< 12:19 0:00 [md]
S 12:19 0:00 [kworker/1:1]
S< 12:19 0:00 [rpciod]
S 12:19 0:00 [kworker/0:1]
S 12:19 0:02 [kswapd0]
S 12:19 0:00 [fsnotifymark] S 12:19 0:00 [ecryptfs-kthrea] S< 12:19 0:00 [nfsiod] S< 12:19 0:00 [cifsiod] S 12:19 0:00 [jfsIO] S 12:19 0:00 [jfsCommit] S 12:19 0:00 [jfsCommit] S 12:19 0:00 [jfsSync] S< 12:19 0:00 [xfsalloc] S< 12:19 0:00 [xfsmrucache] S< 12:19 0:00 [glockworkqueue]
S< 12:19 0:00 [deleteworkqueu] S< 12:19 0:00 [gfsrecovery]
S 12:19 0:00 [khvcd]
S< 12:19 0:00 [bioset]
S< 12:19 0:00 [drbd-reissue]
S< 12:19 0:00 [rbd]
S< 12:19 0:00 [kpsmoused]
S< 12:19 0:00 [raid5wq]
S< 12:19 0:00 [dmbufiocache]
S< 12:19 0:00 [ipv6_addrconf]
S< 12:19 0:00 [ceph-msgr]
S< 12:19 0:00 [bioset]
S< 12:19 0:00 [deferwq]
S< 12:19 0:00 [kworker/0:1H]
S 12:19 0:00 [jbd2/xvda-8]
S< 12:19 0:00 [ext4-rsv-conver]
Ss 12:19 0:00 /usr/sbin/rpc.idmapd
Ss 12:19 0:00 /usr/sbin/atd
S< 12:21 0:00 [kworker/1:1H]
S 14:00 0:00 [kworker/u4:1]

````

forum:mattltm 9 years ago

Ahh!

From that it looks like php-fpm is using lots of resources.

Maybe I should swap the sites over to fast-cgi or mod_php?

forum:obs 9 years ago

Looks like you've quite a lot running on there, clamav is your biggest user with 384MB which isn't unusual it's a memory hog. PHP is your next largest culprit but it's not using an excessive amount. You have multiple PHP-FPM pools which from a security point of view is good since you can isolate sites but from a resource point of view it's bad since each site requires it's own pool.

Changing to mod_php will only make things worse since even static files will require a process that has PHP loaded, fastcgi is fpm's predecessor so no benefit there.

Now you say you're using 100% swap, this might not be a bad thing, how big is the swap? And can you see how much swapping you're doing? Linux will put things into swap that aren't being used very often, which is good since it frees up memory for file caches and other things.

forum:mattltm 9 years ago

The Swap is 512MB.

I've changed all of the sites to use fast-cgi and it seems that the swap is now sitting at around 20% used with the RAM at around 70%.

I'll keep an eye on this over the next few days and see if it's stable.

Thanks.

Reply

Description

Please enter an answer

Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Compute

Storage

Databases

Networking

Developer Tools

Delivery

Security

Services

Industries

Pricing

Community

Engage With Us

Maxed out Swap

6 Replies

Reply

Tips: