Sometimes, you don’t have any inodes left

You notice something is wrong with your system. I’ll just put this error message here for the sake of the Googles:

That stinks.

df -h may show that you have free space on the filesystem in question, but do you have free inodes? Use df -i to answer that question. The next question is, “Where did all my inodes go?” Thankfully, the good people in the stackexchange community are on the case. Here’s a slight modification of the answer for readability:

In my case, I had crazy numbers of small files in the postfix “defer” queue, and in the /var/lib/php5 directory where a certain PHP app was creating a session file for every client who connects. In addition to the inodes, cleaning up all that stuff gave me back about a GB of space on my filesystem, probably due to the default 4K blocks used by several hundred thousand tiny files.

I don’t actually know what the postfix defer queue is for, specifically how it differs from the deferred queue. At this point in time, I don’t really care.

That got rid of the mail junk.

I had earlier reduced the number of times that the /etc/cron.d/php5 script runs, because it’s pretty I/O intensive. But since I have this one app running that actually creates those pesky session files, I increased its run frequency. By default it runs 09,39 * * * * (every hour on the nines and thirty-nines), which should be plenty for most folks.