When a Linux server suddenly reports 99% disk usage, it's tempting to start deleting old home directories, backups, or log files. I almost went down that road too.
In my case, this happened on a Virtualmin server hosting multiple WordPress websites. At first glance, /home looked like the biggest directory, so I checked whether an old user directory could be safely removed.
It turned out that wasn't the real problem at all.
This post shows the exact commands I used to find the real disk hog, without deleting random files or risking a broken website.
Safety first: Don't delete anything until you know exactly what is consuming the space. Especially avoid manually deleting files inside
/var/lib/mysql.
The Situation
The server was almost full.
/dev/sda1 99% used
The first thing I wanted to know was which top-level directories were actually using the space.
I ran:
du -xhd1 /var /home /usr /opt /root 2>/dev/null | sort -h
The result immediately narrowed things down.
12G /home
16G /var
15G /var/lib
1.4G /var/log
At first, /home looked suspicious.
Since I had recently moved one user's home directory to another disk, I also checked whether the old directory was still taking space.
It wasn't.
20K /home/example.old
Twenty kilobytes is basically nothing. So deleting it would free almost zero space.
That meant I needed to keep digging.
Looking Inside /var
The next command was simple.
du -xhd1 /var | sort -h
The output made the next step obvious.
15G /var/lib
1.4G /var/log
Logs were a little large, but not large enough to explain why the disk was almost completely full.
So I searched for large files.
find /var -xdev -type f -size +200M -exec ls -lh {} ; | sort -k5 -h
This is one of my favorite commands when a server suddenly fills up.
The Real Culprit
The result was very clear.
11G /var/lib/mysql/.../clean.ibd
1.8G /var/lib/mysql/.../responses.ibd
664M /var/lib/mysql/.../clean_old.ibd
587M /var/log/proftpd/sftp.log
There it was.
The largest file on the entire server wasn't an old backup.
It wasn't a forgotten home directory.
It wasn't Apache logs.
It was a single MariaDB table that had grown to around 11 GB.
Another table was nearly 2 GB, and there was also an old table taking several hundred megabytes.
Suddenly the mystery was solved.
Don't Delete .ibd Files
If you've never worked with MariaDB or MySQL storage before, you might think:
"I'll just delete the 11 GB file."
Don't.
Files ending in .ibd are InnoDB table files. Deleting them manually will almost certainly corrupt the database.
Instead, find out which tables are using the space.
mysql -e "
SELECT
table_name,
ROUND((data_length + index_length)/1024/1024/1024,2) AS GB
FROM information_schema.tables
WHERE table_schema='your_database'
ORDER BY GB DESC;
"
This safely lists the largest tables.
Once you know what each table is used for, you can decide whether it should be archived, cleaned up, or truncated.
Other Places Worth Checking
Although MySQL turned out to be the real issue on this server, these are still worth checking on almost every Linux web server.
System logs
du -sh /var/log/*
If systemd journals have grown too much, you can safely reduce them.
journalctl --vacuum-time=7d
Package cache
APT caches downloaded packages that are no longer needed.
apt clean
Website backups
Virtualmin or manual backups can quietly consume many gigabytes.
Check your backup directories before assuming the database is the problem.
Lessons Learned
I almost spent time deleting old home directories because /home looked large.
Fortunately, a few simple commands showed the real problem within minutes.
The workflow I now follow is:
- Check top-level disk usage with
du. - Drill into the largest directory.
- Search for files larger than a few hundred megabytes.
- Never manually delete MariaDB
.ibdfiles. - Verify before cleaning anything.
Those three commands quickly narrowed a 99% full disk down to one oversized database table.
Sometimes the biggest directory isn't the actual problem. And sometimes the file you should not delete is exactly the one taking the most space.
Hopefully this saves you from making the same mistake.
Happy troubleshooting!
No Comments