Best way to prevent the root system filling up when a mount fails?

Solution 1:

The most error-proof solution is to make the mount point unwritable. This would be your solution #3. However there is one additional step you should perform. chattr +i /mnt/backups. This is because even with no permissions, root would still be able to write to the directory. With chattr +i (sets immutable flag) not even root can write to it. Once the mount is mounted, the permissions dont matter as the permissions will be of the remote directory, not the local one.

Solution 2:

Number 5 - Put a test in your backup script to ensure that the directory is mounted before continuing. The script should fail if the mount is not available or present. Or you can just make sure things are mounted prior to running the backup.

Try the mountpoint command, which checks if a specified directory is a mountpoint:

mountpoint -q /mnt/backups || mount /mnt/backups

---

Solution 3:

What ewwhite said. Also, some extra monitoring for basis system health wouldn't be a bad idea.

Something like Monit can check to see how much space is left. If you want to go full bore on system monitoring, you can look at Nagios, but Monit is light weight and will do the basics.

Since you're using Ubuntu, Monit is already in repo, so you can do "sudo apt-get install monit", then start looking at the configuration files to tell it to send alerts to the right place, monitor the right services, etc. Here's a quick tutorial.