How to free disk space from deleted but still referenced file

Very rarely and often by a mistake you can end up with deleted file that is still used by some process, which is likely writing to it, so the used space cannot be freed.

The running process is still holding a reference to it, so the used disk space cannot be freed. You can kill this process right away or truncate the file to reclaim used disk space. It will not fix the source of the problem, it is not safe, but can give you enough time to fix this issue properly.

I will describe this specific case on Debian Squeeze, but the whole process is still revelant as of today.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 6.0.10 (squeeze)
Release:	6.0.10
Codename:	squeeze

Notice that glusterfs process is still writing to the deleted log file.

$ sudo lsof -p 7447
COMMAND    PID USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
glusterfs 7447 root  cwd    DIR        8,1      4096      2 /
glusterfs 7447 root  rtd    DIR        8,1      4096      2 /
glusterfs 7447 root  txt    REG        8,1     57792  16030 /usr/sbin/glusterfsd
glusterfs 7447 root  mem    REG        8,1     90504  15601 /lib/libgcc_s.so.1
glusterfs 7447 root  mem    REG        8,1     92640 127616 /usr/lib/glusterfs/3.2.7/xlator/debug/io-stats.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    121472 127590 /usr/lib/glusterfs/3.2.7/xlator/performance/stat-prefetch.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     86960 127586 /usr/lib/glusterfs/3.2.7/xlator/performance/quick-read.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     71744 127588 /usr/lib/glusterfs/3.2.7/xlator/performance/io-cache.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     45416 127577 /usr/lib/glusterfs/3.2.7/xlator/performance/read-ahead.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     71752 127575 /usr/lib/glusterfs/3.2.7/xlator/performance/write-behind.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    393760 127544 /usr/lib/glusterfs/3.2.7/xlator/cluster/afr.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    234976 127571 /usr/lib/glusterfs/3.2.7/xlator/protocol/client.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     80712  30248 /lib/libresolv-2.11.3.so
glusterfs 7447 root  mem    REG        8,1     22928  34763 /lib/libnss_dns-2.11.3.so
glusterfs 7447 root  mem    REG        8,1     51728  34761 /lib/libnss_files-2.11.3.so
glusterfs 7447 root  mem    REG        8,1     76936 127535 /usr/lib/glusterfs/3.2.7/rpc-transport/socket.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    142168 127592 /usr/lib/glusterfs/3.2.7/xlator/mount/fuse.so.0.0.0
glusterfs 7447 root  mem    REG        8,1   1478056  34754 /lib/libc-2.11.3.so
glusterfs 7447 root  mem    REG        8,1    131261  34764 /lib/libpthread-2.11.3.so
glusterfs 7447 root  mem    REG        8,1     14696  34760 /lib/libdl-2.11.3.so
glusterfs 7447 root  mem    REG        8,1     91952  17982 /usr/lib/libgfxdr.so.0.0.0
glusterfs 7447 root  mem    REG        8,1     88584  17739 /usr/lib/libgfrpc.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    400504  17697 /usr/lib/libglusterfs.so.0.0.0
glusterfs 7447 root  mem    REG        8,1    128744  34748 /lib/ld-2.11.3.so
glusterfs 7447 root    0u   CHR        1,3       0t0   2045 /dev/null
glusterfs 7447 root    1u   CHR        1,3       0t0   2045 /dev/null
glusterfs 7447 root    2u   CHR        1,3       0t0   2045 /dev/null
glusterfs 7447 root    3u  0000        0,9         0   2042 anon_inode
glusterfs 7447 root    4w   REG        8,1 993079299 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted)
glusterfs 7447 root    5u   CHR     10,229       0t0   7324 /dev/fuse
glusterfs 7447 root    6u  IPv4 1112791139       0t0    TCP gfs-a.local:1022->gfs-a.local:24007 (ESTABLISHED)
glusterfs 7447 root    7u  IPv4 1095043971       0t0    TCP gfs-a.local:1023->gfs-a.local:24009 (ESTABLISHED)
glusterfs 7447 root    8u   REG        8,1         0  16722 /tmp/tmpf6nhOjW (deleted)
glusterfs 7447 root    9u  IPv4 1235641780       0t0    TCP gfs-a.local:1020->gfs-b.local:24009 (ESTABLISHED)
glusterfs 7447 root   10r   CHR        1,9       0t0   2050 /dev/urandom

Filter the data to display only open files that have been unlinked for specific process.

$ sudo lsof -a +L1  -p 7447
COMMAND    PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK   NODE NAME
glusterfs 7447 root    4w   REG    8,1 993079299     0 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted)
glusterfs 7447 root    8u   REG    8,1         0     0  16722 /tmp/tmpf6nhOjW (deleted)

Log file is cannot be truncated directly as it is not available in the filesystem, but you can access it by using file descriptor opened by this particular process.

$ sudo ls -l /proc/7447/fd
total 0
lrwx------ 1 root root 64 Nov 21 11:02 0  -> /dev/null
lrwx------ 1 root root 64 Nov 21 11:02 1  -> /dev/null
lr-x------ 1 root root 64 Nov 21 11:02 10 -> /dev/urandom
lrwx------ 1 root root 64 Nov 21 11:02 2  -> /dev/null
lrwx------ 1 root root 64 Nov 21 11:02 3  -> anon_inode:[eventpoll]
l-wx------ 1 root root 64 Nov 21 11:02 4  -> /var/log/glusterfs/mnt-backup.log.1.old (deleted)
lrwx------ 1 root root 64 Nov 21 11:02 5  -> /dev/fuse
lrwx------ 1 root root 64 Nov 21 11:02 6  -> socket:[1112791139]
lrwx------ 1 root root 64 Nov 21 11:02 7  -> socket:[1095043971]
lrwx------ 1 root root 64 Nov 21 11:02 8  -> /tmp/tmpf6nhOjW (deleted)
lrwx------ 1 root root 64 Nov 21 11:02 9  -> socket:[1235641780]

Display file descriptors pointing to deleted files in a very simple way.

$ sudo find /proc/7447/fd -ilname "*(deleted)"
/proc/7447/fd/4
/proc/7447/fd/8

Display file descriptors pointing to deleted files in a more useful way.

$ sudo find /proc/7447/fd  -ilname "*(deleted)" -printf "%h/%f -> %l\n"
/proc/7447/fd/4 -> /var/log/glusterfs/mnt-backup.log.1.old (deleted)
/proc/7447/fd/8 -> /tmp/tmpf6nhOjW (deleted)

Display file descriptor pointing to the particular deleted file.

$ sudo find /proc/7447/fd  -lname "*mnt-backup.log.1.old (deleted)"
/proc/7447/fd/4

Truncate this file descriptors.

$ sudo truncate  /proc/7447/fd/4 --size 0

Filter the data to display only open files that have been unlinked for specific process.

$ sudo lsof -a +L1 -p 7447
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
glusterfs 7447 root    4w   REG    8,1      729     0 127832 /var/log/glusterfs/mnt-backup.log.1.old (deleted)
glusterfs 7447 root    8u   REG    8,1        0     0  16722 /tmp/tmpf6nhOjW (deleted)

This will give you some time.

Milosz Galazka's Picture

About Milosz Galazka

Milosz is a Linux Foundation Certified Engineer working for a successful Polish company as a system administrator and a long time supporter of Free Software Foundation and Debian operating system.