Is there any other reason for "no space left on device"?

Solution 1:

Looking at the 2% of inodes left made me think about the root reserves that the EXT filesystem imposes. You may want to check these out:

1. "Reserved space for root on a filesystem - why?"
2. "Reasonable size for “filesystem reserved blocks” for non-OS disks?"

I would try to .tar.gz some of the older backups hoping that it would reduce the number of inodes in use.

Solution 2:

My suspicion (see EDIT3) apparently was right: Adding acl support to the file system made rsync/dirvish think that all the files had changed. So instead of making an incremental backup and just create hard links to the already existing files, it tried to create a full backup which of course failed because the hard disk did not have enough space for that.

So the error message was actually correct.

After starting again with an empty backup disk, the incremental backups worked as before.

---

Solution 3:

I see that dummzeuch find a solution to his problem but there is actually one more case I found where disk can have enough inodes/free space and still showing "no space left on the device" while attempting to transfer certain directories.

This is caused by hash collisions on block devices formatted with ext4 file system where directory indexing is enabled too especially where single directory hosts more than 100k files in it and name of the files are generated from the same algorithm (cache files, md5sum file names etc.)

Solution is to try with another directory indexing algorithm:

tune2fs -E "hash_alg=tea" /dev/blockdev_name

or to disable completely directory indexing for that block device (may hurt performance)

tune2fs -O ^dir_index /dev/blockdev_name

Another solution is to see what is filling the directory with such files and fix the software.

Possible solution is split content of folder with huge volume of files in it to multiple separate subfolders.

Full description of the problem is presented by Axel Wagner here

http://blog.merovius.de/2013/10/20/ext4-mysterious-no-space-left-on.html

Cheers.