No space left on device even though there is

If you run into this problem, where you get errors that seem to indicate that the disk is full when it's not, make sure to also check the inode utilization.

You can use df -i to get a quick report on the used/available inodes for each mount point.

If you see that you are running very low, or out of, inodes then the next step is to identify which folder is holding up most inodes. Since each file and directory uses an inode, you could have a folder with hundreds of thousands of tiny, or empty files that are using up all the inodes. Usual suspects include: temp directory, website cache directories, package cache directories etc.

Use this command to get an ordered list of the subdirectories with the most inodes used:

sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

Run this in your root folder, then drill down until you find your culprit.

At some point in the past, your root filesystem filled up, and a small, temporary /tmp was created to allow boot to succeed. This small /tmp was never deleted, so now, even though you have room on /, you still are filling up the small /tmp and seeing your problem. Simply unmount it:

sudo umount /tmp

and of course, try to ensure your / is as clean as possible.

---

Normally, /tmp is just a part of the root (/) filesystem, no separate mount is needed, unless there are special circumstances, like running out of root filespace (when some daemon creates the one you see), or maybe you have / on a very slow media (like an USB flash stick) and want /tmp in ram for performance, even with limited space.

I believe you have a lot of unused files remove them with:

sudo apt autoremove

Then re-check your space with df command