While it's usually clear if your system is running low on memory or using too much CPU time, disk usage is another key metric that can quietly affect you over time if you leave your server unattended. You need to regularly check your disk usage with these commands.
Check Disk Usage on Linux
A utility used to quickly check disk usage on almost any Linux system is df, which stands for "disk filesystem". It just prints out a list of all filesystems on the system.
df -hT
The command here is invoked with two flags, -h for "human readable", which prints out the number of bytes in KB, MB, and GB, and -T to show the type of filesystem.
df will print out the size of each filesystem, how many are used and available, and where it is mounted on the system.
You'll quickly notice that you may have more "filesystems" than you expected. The server has only one solid-state drive, but more than 20 file systems. Most of these are backend stuff for other programs and services, such as compressed squashfs for containers, virtual tmpfs, and Docker systems. overlay
In this example, ext4 is the real drive, which is obvious since it is mounted at the root, but may not be wiped out immediately on a multi-drive system. -t If desired, this list can be filtered by specifying the types to look at using the lowercase flag:
df -hT -t ext4
-x or manually remove what you don't want to see by using the flag:
df -hT -x squashfs -x overlay -x tmpfs -x devtmpfs
You can also ask df for information about any specific filesystem, and even include wildcards to match multiple systems by name:
df -h /dev/md*
Alternatively, you can ask it for information about a specific mount, which is most useful for quickly getting information on the root system:
df -h /
Monitoring at a glance
But most of the time, you probably don't want to remember and type a bunch of commands with specific flags. That's what the glances utility is for, and we highly recommend you try it.
top It basically replaces the built-in utilities htop and etc, except that it shows a lot of different performance metrics about your system (i.e. disk). It's not included by default in most Linux distributions, but it's open source and can be installed from pip since it's a Python application.
sudo pip install glances
Then just run the application to open the glances dashboard:
glances
In the lower left corner you'll find some information about disk usage, including current I/O speed and total usage per physical disk. Using it, you can easily spot disk filling problems before they corrupt your system.
There are many other utilities that can be used to monitor your system, but glances checks all the boxes, so it's a really good tool.
Send alerts when your disk usage is too high
The main problem with command line tools is that they require you to actively check for problems. But problems can crop up suddenly, so it's best to be informed in advance.
So the solution was to set up an automatic cron-running daily job, df, to check the usage on the root system. It will compare to the set value and if it is greater it will send a message.
#!/bin/bash
CURRENT=$(df / | grep / | awk '{ print $5}' | sed 's/%//g')
THRESHOLD=90
if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"Your server \`$(hostname)\` is currently at ${CURRENT}% disk capacity.\"}"
fi
You have many options for how to get messages, depending on your preferred contact. The easiest way is to set up the mail utility to send you email from the command line. You can read our article on setting up Postfix to handle this for you.
A cooler approach is to message yourself directly on your active messaging platform, such as setting up a Slack notification from your server, which can easily be done using a webhook with a curl POST request.
