How to "yum update" / "aptitude safe-upgrade" many hosts?

Puppet is, really, just a configuration management utility, not an automation tool. If you want proper automation, then you'll want to be looking at mcollective which started out as a 3rd party tool, and has now been brought under the umbrella of puppetlabs. Not having worked with mcollective, I can't really speak to how well it would handle this either, but my understanding is that it works best as an arbitrary task execution mechanism, which won't work as well when attempting to do regularly repeated tasks.

I believe the best way to do this would be to develop a the scripts and process by which you want to update, and then use puppet to push out the configs. So ask yourself the following questions.

1. How often do I want the machines to update?
2. Do I want them to automatically reboot when a new kernel is installed, or do I just want a notification?
3. Should we be running any special commands during updates?
4. Do I have enough systems that updates should be staggered?

As you start building out the config, and answering those questions, you will likely find more that will come out of your specific environment. For a specific example what I do is this:

• For most of my systems updates are once per week, via cron job. Depending on purpose and environment, some systems are much more frequently.
• Most systems reboot automatically, certain systems (depending on purpose) email out a reminder to reboot. This is handled through a cron job that checks the highest version numbered vmlinuz file in /boot against the output of uname -r
• After getting burned by the glibc update from RHEL 5.3->5.4 I make sure to update yum, then glibc, then everything else.
• Since all this is run by cron jobs, I have sleeps with random times in between each yum run. Each host can take between 5 minutes and 45 minutes to complete, which does a reasonable job of spreading the load.

This is all contained within two files, the two cron entries (updates and kernel check) stored in /etc/cron.d and the update script. I'm doing this with an increasingly messy looking shell script that takes command line arguments to perform the update or kernel check and whether or not to reboot. Puppet then manages these two files. Since you're dealing with both rpm and dpkg based systems, I would probably create either two scripts or make separate do_update functions for the two flavors and have each called with a different command line switch. Then you can either push out the right script by checking the operatingsystem fact in your file declaration, or templatize the cron entry and use the same fact to decide which switch to use.

Using a well tested script, this method has worked out very well. Well enough, in fact, that this setup has been used successfully for a few years to handle hundreds of systems. We will occasionally see hiccups when the kernel packagers start adding more and more levels of minor versions to the files and the comparison routine chokes.

Puppet itself is not the tool for this job. Puppetlabs has a seperate tool called MCollective, which is similar to Capistrano, Fabric and Func.

There is an MCollecive agent called the Packages Agent which is able to do yum updates, among other types of package management.

You mentioned that part of your infrastructure is running Ubuntu. Perhaps you should look into the unattended-upgrades package.

If you're hosts are RHEL hosts you may wish to look into RHN Satellite. Otherwise Spacewalk may be worth a look **(see note below). They are designed to manage local copies of upstream package repositories and to facilitate the management of systems registered to the RHN Satellite server. You can schedule for packages to be updated at a specific time, or if you have enabled OSAD the updates can be scheduled in neartime in the RHN Satellite/Spacewalk Web GUI.

Some people have posted using puppet below, there are some blogs and web pages written up about how to integrate Puppet with RHN Satellite.

ALTERNATIVELY....

If you're more interested in looking at an emerging project which will eventually replace RHN Satellite you can have a look at the Katello Project. Katello contains a lot of projects which are part of Red Hat's CloudForms product and is said to be the upstream for RHN Satellite's eventual replacement. In fact Katello integrates Puppet via the use of Foreman. Looking long term Katello is definitely worth considering over Spacewalk (but not RHN Satellite yet due to subscription/support).

** I have referred to both RHN Satellite and Spacewalk as RHN Satellite in my post to make things easier. The real distinction comes to weather or not you have RHEL systems, if that is the case you will need RHN Satellite as it can download from RHN in a supported manner, Spacewalk would be for users using a Rebuild of RHEL or Fedora (I understand Spacewalk may support Debian as well, but I do not know much about this personally.) Thus when searching for information it may be necessary to search for Spacewalk and or RHN Satellite