How Often Should I Update our Linux Server?
Solution 1:
I run apt-get update -qq; apt-get upgrade -duyq daily. This will check for updates, but not do them automatically.
Then I can run the upgrades manually while I am watching, and can correct anything that might go wrong.
Besides the security concerns of maintaining a patched system, I find that if I leave it too long between patches, I end up with a whole bunch of packages that want to be upgraded, and that scares me a-lot more than just upgrading one or two every week or so. Therefore I tend to run my upgrades weekly, or if they are high priority, daily. This has the added advantage of knowing which package broke your system (ie. if you're only upgrading a couple at a time)
I always upgrade less critical systems first. I also have a "rollback plan" in place in case I can't fix the system. (since most of our servers are virtual, this rollback plan usually consists of taking a snapshot before the upgrade that I can revert to if necessary)
That being said, I think an upgrade has broken something only once or twice in the past 4 years, and that was on a highly customized system - so you don't have to be TOO paranoid :)
Solution 2:
On top of previous answers - a couple more specifically Debian things: you should Subscribe to debian-security-announce and debian-announce and / or check out the Debian Security page.
Solution 3:
Assuming you are running the stable release of Debian, most patches will be security or bug related which should mean that there won't be too many major changes between versions of any given package. According to the debian patch policy patches should also have been in testing for some time before being moved to the stable branch by the maintainer. Obviously This won't stop breakages when patching but should prevent them in most cases.
It would be prudent to ensure that your testing sever is kept upto date and any packages that have bugs affecting you and your servers should be kept up to date. All packages that have security advisories against them should be updated as soon as you know the patch is stable.
Debian is usually a very stable OS and not one you should be overly concerned with breakages on however always read what is going to be updated before it get updated and keep an eye out for anything that seems strange. I use VCS on my /etc/ dir as well to ensure that any config file changes can be seen with a 'git diff' command.
Solution 4:
I do a dry run (first) to see what is going to be updated. Sometimes, libraries (lets call it libfoo for this example) change their API, which breaks programs that we wrote / installed ourselves. If some critical library is updated, I grab the source and try rebuilding our stuff against it prior to updating.
I also check to see that we're not jumping to an intermediate version of some public service, ie apache, etc. I'd rather stay a year behind and not encounter random breakage, unless the update is critical.
If you are a system administrator, you should be pulling RSS feeds from sites like Secunia, which should let you know way ahead of time if your distro is going to be pushing some patches.
Do not ever, ever just blindly upgrade / update. Unfortunately, the task of knowing what is broken falls on you, not your distro package manager, especially if your systems support programmers.
Solution 5:
Where I work we have a pretty extensive process that involves using software called PatchLink to notify us of the most important security related updates, and we apply them after testing, on a package by package basis. We have thousands of servers though.
If you only have two servers the process should be much simpler. Although I do not think doing an "apt-get update/upgrade" is your best bet.
I would monitor patches for the software you are running, and make decisions based on the fixes in those releases on when to upgrade.
Since you have a test server, obviously, always test the update before applying them.