We faced the problem mentioned in the title, and we found out the solution. We are more than glad to share it here with you. Enjoy the reading, check the rest of our blog posts, and may the power of our solutions be with you.
Error?
Do you know the feeling when your server starts to behave in a strange way, don’t you? So, you spend hours debugging, as we did some time ago, just to find out the “No Space Left On Device” error? It doesn’t matter what you do; the error keeps to pop out. If your disk is really full, then it is an easy problem to solve. Just clean it up. But, if your disk is not full the issue becomes a little bit more complicated… but still easy solvable. It’s possible that you have run out of inodes.
What are inodes?
Wikipedia says:
The inode (index node) is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory. Each inode stores the attributes and disk block location(s) of the object’s data. File-system object attributes may include metadata (times of last change, access, modification), as well as owner and permission data.
What does it mean in real life?
Files are not only using up your disk space, but they are also using disk inodes which hold information about them. In practice, if you keep a lot of big files most likely, you will run out of space. But, if you have a lot of very small files most likely, you will run out of inodes. Even though, there will be still potential free space on your drive.
How to check the number of inodes used?
Long story short, to check drive space we always use:
df -h
Thus, to check the number of inodes just put:
df --inodes
Do you know what happened?
The output is a little bit different. We use only 38% of the disk space but already 62% of the disk inodes available.
How to identify files and folder consuming your inodes?
The last key question to be answered is how you can identify where those files that consume your inodes are hiding. We use a very simple command for it:
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
As an output, you will receive a list of folders and the number of inodes that are being used by them.
Now you just have to navigate to the right place and clean it up 🙂
How about trying hosting services with unlimited disk space?
In case of any questions or problems, contact us.
Nice trick, but it doesn’t work when your boot volume inodes are full, as /tmp cannot create the files needed to use sort.
In this case, if you have another disk with space and inodes available, you can get around this as follows:
mkdir /otherdisk/temptmp
export TMPDIR=/otherdisk/temptmp
Then the commands should work as above.
Great tip!!!
Great tip. I ran in to this same issue and because of your post, I was able to solve it and get past this issue. Had been struggling for a while to figure out what was wrong.
Cool!!!!
The solution has not been solved, I’m having a problem when the plugin has an error
Hello, can you explain which plugin? Does it have anything to do with lack of disk space?
We got this error when our disk space use is only 67% and inodes is about 3% only. Also we got this error as isolated as the batch ran on its next schedule run was completed without error. What might be the cause of this isolated issue?
It is also difficult to say without accessing the system, are you sure that there wasn’t any other partition full?
Could we increase the inodes limit?(Sorry if the question is mad.)
Yes you can 🙂 https://unix.stackexchange.com/questions/26598/how-can-i-increase-the-number-of-inodes-in-an-ext4-filesystem