disk space keeps getting 100% full
Today, my server's / partition was full.
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 4030144 3866588 0 100% /
/dev/xvdc 11216120 1109484 9992688 10% /home
/dev/xvdd 1007896 19152 978504 2% /tmp
Size in / partition is listed below :-
root@server [/]# du -h –max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.7M ./tmp
26M ./lib
360M ./var
4.3M ./root
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./
I logged into linode manager and increased the / size to 5 GB and it fixed the problem.
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 2947572 2009100 60% /
/dev/xvdc 10077504 1109484 8968020 12% /home
/dev/xvdd 1007896 19096 978560 2% /tmp
But after a few hours, when I checked, the disk space was again full. See below :-
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 4957028 0 100% /
/dev/xvdc 10077504 1109476 8968028 12% /home
/dev/xvdd 1007896 19132 978524 2% /tmp
though nothing has changed in terms of size.
see the disk space usage again,
root@server [/]# du -h –max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.7M ./tmp
26M ./lib
356M ./var
4.3M ./root
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./.spamassassin
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./
Finally, I had to reboot the system once again to fix this :
root@server [~]# w
15:24:42 up 1 min, 1 user, load average: 0.52, 0.27, 0.11
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root ttyp0 61.17.17.91 15:24 0.00s 0.02s 0.00s w
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5166712 2971412 1985260 60% /
/dev/xvdc 10077504 1109476 8968028 12% /home
/dev/xvdd 1007896 19132 978524 2% /tmp
It is again happening with no apparent change in disk. If I reboot the system it will be alright. Is anybody else facing the same issue? Can anyone help here?
9 Replies
Process 1 opens a file.
Process 2 deletes the file from the filesystem so it can't be seen by ls, du, or the normal tools. df will see the space used by this hidden file. This does not actually free the space used by this file until process 1 closes it, it's still on disk just most tools don't see it.
Process 1 keeps writing to the file unaware that it was deleted by process 2. It's still a totally valid file from the point of view of process 1.
My guess would be that process 1 is syslog and process 2 is logrotate but they could be any other processes that write and remove files. Homebrew logfile handling scripts are notorious for causing these kind of problems.
ls -l /proc/*/fd should help you find out what process 1 is.
Another less likely thing that could be happening is a process is opening and deleting a file so it can use the resulting hidden file as a kind of temp storage. It could then be putting way to much data in there. Some processes use deleted files as a temp working area.
root@server [/]# du -h –max-depth=1 ./
0 ./sys
8.4M ./etc
4.0K ./selinux
25M ./opt
5.1M ./scripts
956M ./home
4.7M ./bin
24M ./sbin
2.8M ./tmp
26M ./lib
401M ./var
4.4M ./root
du: `./proc/1': No such file or directory
du: `./proc/2': No such file or directory
du: `./proc/3': No such file or directory
du: `./proc/4': No such file or directory
du: `./proc/149': No such file or directory
du: `./proc/233': No such file or directory
du: `./proc/235': No such file or directory
du: `./proc/271': No such file or directory
du: `./proc/272': No such file or directory
du: `./proc/273': No such file or directory
du: `./proc/274': No such file or directory
du: `./proc/278': No such file or directory
du: `./proc/280': No such file or directory
du: `./proc/283': No such file or directory
du: `./proc/284': No such file or directory
du: `./proc/285': No such file or directory
du: `./proc/286': No such file or directory
du: `./proc/287': No such file or directory
du: `./proc/288': No such file or directory
du: `./proc/289': No such file or directory
du: `./proc/290': No such file or directory
du: `./proc/291': No such file or directory
du: `./proc/292': No such file or directory
du: `./proc/293': No such file or directory
du: `./proc/852': No such file or directory
du: `./proc/855': No such file or directory
du: `./proc/945': No such file or directory
du: `./proc/971': No such file or directory
du: `./proc/1001': No such file or directory
du: `./proc/1023': No such file or directory
du: `./proc/2961': No such file or directory
du: `./proc/2962': No such file or directory
du: `./proc/2968': No such file or directory
du: `./proc/2969': No such file or directory
du: `./proc/2974': No such file or directory
du: `./proc/2975': No such file or directory
du: `./proc/2980': No such file or directory
du: `./proc/2981': No such file or directory
du: `./proc/3291': No such file or directory
du: `./proc/3339': No such file or directory
du: `./proc/3376': No such file or directory
du: `./proc/3378': No such file or directory
du: `./proc/3387': No such file or directory
du: `./proc/3475': No such file or directory
du: `./proc/3482': No such file or directory
du: `./proc/3504': No such file or directory
du: `./proc/3513': No such file or directory
du: `./proc/3514': No such file or directory
du: `./proc/3541': No such file or directory
du: `./proc/3543': No such file or directory
du: `./proc/3544': No such file or directory
du: `./proc/3608': No such file or directory
du: `./proc/3959': No such file or directory
du: `./proc/4057': No such file or directory
du: `./proc/7267': No such file or directory
du: `./proc/12794': No such file or directory
du: `./proc/12890': No such file or directory
du: `./proc/12892': No such file or directory
du: `./proc/13868': No such file or directory
du: `./proc/16815': No such file or directory
du: `./proc/16959': No such file or directory
du: `./proc/16960': No such file or directory
du: `./proc/16961': No such file or directory
du: `./proc/16962': No such file or directory
du: `./proc/16963': No such file or directory
du: `./proc/16964': No such file or directory
du: `./proc/22942': No such file or directory
du: `./proc/24010': No such file or directory
du: `./proc/24016': No such file or directory
du: `./proc/24017': No such file or directory
du: `./proc/24018': No such file or directory
du: `./proc/28621': No such file or directory
du: `./proc/28803': No such file or directory
du: `./proc/28805': No such file or directory
du: `./proc/28842': No such file or directory
0 ./proc
2.4G ./usr
4.0K ./srv
4.0K ./mnt
4.0K ./.spamassassin
4.0K ./boot
16K ./lost+found
40K ./dev
4.0K ./media
3.8G ./
root@server [/]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 4988440 0 100% /
/dev/xvdc 10206520 1110084 8992728 11% /home
/dev/xvdd 1007896 19176 978480 2% /tmp
root@server [/]# w
After reboot :
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 2985716 2000780 60% /
/dev/xvdc 10206520 1110084 8992728 11% /home
/dev/xvdd 1007896 19144 978512 2% /tmp
I tried the ls -l /proc/*/fd
the result is in the link here :-
Any luck on how to solve this? Within minutes of posting this reply, the disk usage has become like this,
root@server [~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 3632084 1354412 73% /
/dev/xvdc 10206520 1110132 8992680 11% /home
/dev/xvdd 1007896 19196 978460 2% /tmp
root@server [~]#
@visiondream3:
I tried the ls -l /proc/*/fd
the result is in the link here :-
http://leelahost.com/tmpfiles/tmp.html
Yikes, that's long!
De-HTMLing it (yuck) and grepping for "(deleted)":
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
lrwx------ 1 root root 64 May 13 14:44 12 -> /tmp/ib8JF7Hq\ (deleted)
lrwx------ 1 root root 64 May 13 14:44 5 -> /tmp/ibcGKxZ5\ (deleted)
lrwx------ 1 root root 64 May 13 14:44 6 -> /tmp/ib3ZP57a\ (deleted)
lrwx------ 1 root root 64 May 13 14:44 7 -> /tmp/ibwhaEgg\ (deleted)
lrwx------ 1 root root 64 May 13 14:44 8 -> /tmp/ibbd77pl\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:44 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
lrwx------ 1 root root 64 May 13 14:48 12 -> /tmp/ib8JF7Hq\ (deleted)
lrwx------ 1 root root 64 May 13 14:48 5 -> /tmp/ibcGKxZ5\ (deleted)
lrwx------ 1 root root 64 May 13 14:48 7 -> /tmp/ibwhaEgg\ (deleted)
lrwx------ 1 root root 64 May 13 14:48 8 -> /tmp/ibbd77pl\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
l-wx------ 1 root root 64 May 13 14:48 12 -> /usr/local/apache/logs/ssl_mutex\ (deleted)
The /usr/local/apache ones seem to be Apache (duh); the /tmp ones are MySQL.
Regarding actual answers, I dunno…
(caker's "lsof | grep '(deleted)'" is better, BTW. It reports the name of the process and the size of the file.)
# lsof | grep "(deleted)"
apache2 432 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 432 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 796 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 796 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 915 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 915 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 916 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 916 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 1001 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 1001 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 1157 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 1157 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 1287 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 1287 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 1299 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 1299 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 9310 www-data 2w REG 202,0 16049417 43253 /var/log/apache2/error.log.1 (deleted)
apache2 9310 www-data 11w REG 202,0 217594 43255 /var/log/apache2/other_vhosts_access.log.1 (deleted)
apache2 9310 www-data 12w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 13w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 14w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 15w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 16w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 17w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 18w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 19w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 20w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 21w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 22w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 23w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 24w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 25w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 26w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 27w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 28w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 29w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 30w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 31w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 32w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 33w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 34w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 35w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 36w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 37w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 38w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 39w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 40w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 41w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 42w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 43w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 44w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 45w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 46w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 47w REG 202,0 28194695 41355 /var/log/apache2/access.log.1 (deleted)
apache2 9310 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 9310 www-data 49u REG 202,0 0 169 /tmp/.xcache.0.0.846930886.lock (deleted)
apache2 14186 root 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 14186 root 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
mysqld 20694 mysql 5u REG 202,0 0 7611 /tmp/ib0Y84AP (deleted)
mysqld 20694 mysql 6u REG 202,0 21 7612 /tmp/ibQBz5Vu (deleted)
mysqld 20694 mysql 7u REG 202,0 0 7613 /tmp/ibU5rgha (deleted)
mysqld 20694 mysql 8u REG 202,0 0 7614 /tmp/ibwpy51P (deleted)
mysqld 20694 mysql 12u REG 202,0 0 7615 /tmp/ibglE9iC (deleted)
apache2 29206 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 29206 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
apache2 30618 www-data 48w REG 202,0 0 24632 /var/run/apache2/ssl_mutex (deleted)
apache2 30618 www-data 52u REG 202,0 0 1485 /tmp/.xcache.0.0.1125898167.lock (deleted)
Is there something I should do about this?
root@server [/]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda 5037696 4986004 492 100% /
/dev/xvdc 10206520 1110188 8992624 11% /home
/dev/xvdd 1007896 19196 978460 2% /tmp
root@server [/]# lsof | grep '(deleted)'
httpd 1109 root 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1110 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1112 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1113 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1114 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1745 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1754 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1755 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1759 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1760 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 1952 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
mysqld 2961 mysql 5u REG 202,48 0 12 /tmp/ibTI2DAX (deleted)
mysqld 2961 mysql 6u REG 202,48 0 13 /tmp/ibgEXiC3 (deleted)
mysqld 2961 mysql 7u REG 202,48 0 14 /tmp/ibBvrYD9 (deleted)
mysqld 2961 mysql 8u REG 202,48 0 15 /tmp/ibCaTSHf (deleted)
mysqld 2961 mysql 12u REG 202,48 0 16 /tmp/ibPVCIXl (deleted)
httpd 3504 root 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
jsvc 3675 root 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)
jsvc 3678 tomcat 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)
httpd 7068 nobody 10w REG 202,0 73 196612 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 7068 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 7068 nobody 17u REG 202,0 384 62317 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 7068 nobody 18u REG 202,0 1 62322 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)
httpd 25982 nobody 10w REG 202,0 35 196620 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 25982 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 25982 nobody 17u REG 202,0 384 62465 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 25982 nobody 18u REG 202,0 1 62466 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)
httpd 30956 nobody 10w REG 202,0 2603 139572 /usr/local/apache/domlogs/palazhiengineering.in-bytes_log.bkup (deleted)
httpd 30956 nobody 12w REG 202,0 0 61097 /usr/local/apache/logs/ssl_mutex (deleted)
httpd 30956 nobody 17u REG 202,0 384 61831 /usr/local/apache/logs/jk-runtime-status.3504 (deleted)
httpd 30956 nobody 18u REG 202,0 1 61869 /usr/local/apache/logs/jk-runtime-status.3504.lock (deleted)
I think I found out what is causing this :
jsvc 3675 root 1w REG 202,0 2051723331 61873 /usr/local/jakarta/apache-tomcat-5.5.30/logs/catalina.out.bkup (deleted)
It is still locking up around 2 GB of disk space and a restart of tomcat seems to have cleared it.
I have now setup a script to stop and start tomcat everyday midnight, which should resolve this issue permanently.
