How do you apply development practices like version control, testing and continuous integration/deployment to system administration?

Solution 1:

Disclaimer: I am one of the developers of Puppet.

The obvious way is to just apply the concepts: define a cycle of development/test/production, and push changes through them. Use version control to track systems.

In short order, starting down that path leads to discovering that you really want tools that automate those things - essentially, that you want to automate systems administration, so that you don't use those techniques on machines, you use them on the system that manages the machines.

Tools like Chef, Puppet, Salt, and CFEngine are all popular tools to address that second need. They work in the general direction of turning systems administration into a central solution that you can version control and test.

The DevOps movement is another source of good information on how to do this. While the precept is a better cooperation between Developers and Operations staff, it also tends down this same direction.

Solution 2:

Short answer is "OS deployment management", "Configuration Management" and "Software Packaging". Long answer follows.

I'd like to add to Daniel Pittman's reply with a breakdown of what forms a "system" in systems administration.

A system or an environment is going to comprise of:

• Servers
• Operating System
• Configuration
• Vendor packages; and
• Local packages

Enveloping these will be processes such as:

• OS deployment or imaging
• Configuration management
• Software package management
• Auditing/logging
• Monitoring
• Backups

And you'd like to have these combine together to assist you with achieving non-functional goals such as:

• Repeatability
• Maintainability
• Measurability
• Performance
• Trackability
• Testability
• Changeability

This is a quick brain dump. I am sure more could be added to all lists.

Your question touches on a number of these without using the specific words. For example, you want to be able to deploy easily and revert back i.e. want maintainability; you want to do it in a test environment and test till it passes i.e. repeatability, testability and measurability; you're thinking of putting vm images in version control because you'd like repeatability of OS and configuration deployments.

There are plenty of tools to assist you with this, some of them mentioned by Daniel. Some others are:

• Kickstarts (RedHat based), Preseed (Debian based), WDS (MS Windows) for deploying known OS environments
• Spacewalk/Satellite (RedHat based), Group Policies (MS Windows) for configuration and package management
• YUM and APT packaging systems for generating, deploying, upgrading and removing packages (sets of binaries, data and configuration comprising a piece of software)
• Nagios, OpenNMS and SCOM for monitoring
• Amanda, Bacula and Windows Backup Server for backups
• Munin, PCP and Hyperic for performance monitoring
• CVS, SVN, GIT or Bazaar for Version Control
• Hudson and Jenkins for build management
• Selenium and Robot for testing
• Bugzilla, Request Tracker and Jira for recording, communication and tracking

Again, this isn't a comprehensive list, but something that I keep in my head to guide me and hopefully it'll help you out too.