VMware ESXi shutdown triggered by APC UPS connected via USB

Solution 1:

Yes, it's possible. Here are details of my similar setup.

Hardware configuration: APC Smart-UPS 1500 connected to the ESXi 5.1 Host via USB. A Linux virtual machine running on this ESXi host. UPS is connected to this VM using ESXi USB pass through option.

Software configuration: NUT (Network UPS Tools) master running in the VM, and native ESXi NUT slave running on the ESXi host.

Shutdown logic: VM is running the UPS driver usbhid-ups which is responsible for the communication with UPS via USB. The upsd process connects to the UPS through the usbhid-ups driver and monitors the UPS state. The upsmon master process running on the same machine connects to the upsd and initiates the shutdown. ESXi host is running the 2nd instance of upsmon which also connects to the same VM upsd via internal network.

On power failure the following sequence takes place:

  1. UPS via usbhid-ups reports to upsd about power failure.
  2. (optional, useful if you want to shutdown in few minutes instead of Low Battery) upsmon on the VM initiates upssched 5 minutes timer. Timer is aborted if power is restored.
  3. When timer fires or when UPS reports Low Battery, the upsmon raises the FSD (forced shutdown) flag to upsd.
  4. In a stand-alone NUT configuration the FSD flag would shutdown the machine. But here the shutdown command is replaced by simple logging like "I should shutdown now but I am waiting for the host instead". And does nothing.
  5. The FSD flag is also read by ESXi upsmon, which initiates the ESXi host shutdown.
  6. ESXi host shuts down all virtual machines one by one. The important thing is that VM which runs the upsd should be shutdown last (using ESXi startup/shutdown sequence configuration).
  7. Important: this VM must have vmware tools installed. When it receives the guest shutdown command from the host, the vmware-tools shutdown script is being started. This script checks for the /etc/killpower flag. If no flag, it does nothing (this means user activated linux shutdown, not the UPS event). But if the flag exists (FSD active), then this script sends to UPS the delayed powerdown command (say, in 3 minutes).
  8. After running vmware-tools script the guest VM shuts down.
  9. ESXi sees the last VM poweroff state and goes down itself (it takes around 1 minute because there is no other machines running now).
  10. In 2 remaining minutes the UPS cuts off the power.
  11. When power is restored, the ESXi starts and powers on all VMs. The UPS monitoring machine must be started first (the same configuration as for shutdown order).

Downloads:

The NUT for Linux could be installed from package.

The native NUT client for ESXi server can be downloaded using last link on this page: http://www.networkupstools.org/download.html

Some my scripts and conf files are here (only changed lines are shown): http://pastebin.com/KkEeanK1

Notes:

Of course there are more details, and it took some time for me to make this working as it should. But now it performs very nicely. This system accounts for the cases when you just shutdown the monitoring VM from inside (vmware-tools script is not run), or if it's a ESXi host initiated VM shutdown (no /etc/killpower flag, so no UPS load off), or if it's a ESXi shutdown (the same). The only important is to have this VM running ASAP after host boot, and shutdown it last (so host down time is predictable - as said above, it is around 1 minute for me and 2 more minutes I reserve just in case).

My UPS monitoring Linux VM is also Samba/NFS sharing server for backup storage, the NAT/DHCP server for VMs, and some other light-weight services. It takes around 22MHz of ESXi CPU shares and around 10MB of active RAM when idle. Due to using the NUT you can power more devices from the same UPS if required, and all they can be shut down gracefully. No PowerChute and/or expensive Network Monitor Card is required.

Solution 2:

Super question. It is actually possible to do this quite nicely - at least on some setups. I have tried the following recipe on a number of ESXi 5.5 hosts. Basically, the solution goes like this:

  1. Enable SSH access on your ESXi host
  2. Create a Linux VM - I use Ubuntu. You only need a very minimal setup - no GUI or anything.
  3. Connect your APC device via USB to the ESXi host and pass it through to the Linux VM.
    • Make sure that the USB controller you add to the VM matches the actual, physical USB controller the APC device is connected to, i.e. only add an XHCI controller if the physical device is a USB3 device. Mismatches seem to cause odd problems in the Linux USB device driver.
    • If things aren't working out and you see errors like ctrl urb status -62 in dmesg, chances are the physical controller doesn't match the one in your VM. If they do match - well then it's a problem. I have one setup with this sort of problem and no real solution to it.
  4. Install apcupsd on the Linux VM - in Ubuntu, you can do sudo apt-get install apcupsd to install the latest version. The NUT project is also nice but I am a traditionalist.
  5. Install the plink utility by doing sudo apt-get install putty-tools
  6. Connect to your ESXI host by doing plink root@<your ESXi host IP>. You can close the connection immediately. The objective is to get the host key saved so plink won't prompt for it again when we run it via a script
  7. Edit /etc/apcupsd/apcupsd.conf and change the items below so they match: UPSNAME < the name you'd like your UPS to have > UPSCABLE usb UPSTYPE usb # DEVICE DIRECTIVE should be blank for USB DEVICE Also make sure that /etc/default/apcupsd has ISCONFIGURED=yes
  8. Edit /etc/apcupsd/apccontrol and scroll to the doshutdown case. Make it look like this: doshutdown) echo "UPS ${2} initiated Shutdown Sequence" | ${WALL} # Shut down indirectly by triggering the ESXi host to do the # shutdown via VMWare tools /usr/bin/plink root@< your ESXi host IP > -pw < your root pw > "/sbin/shutdown.sh && /sbin/poweroff" ;;
  9. Restart apcupsd using sudo service apcupsd restart and see if things are working by invoking apcaccess. If not, check logs and dmesg
  10. Make sure all VMs that need to shut down nicely in case of a power failure have VMWare Tools installed. Also make sure that they are part of the VM startup/shutdown list (in the vSphere Web Client, go to: vCenter -> <your host> -> Manage -> Settings -> VM Startup/Shutdown). Make sure that the shutdown action is to shut down the guest OS.

Once you have these things running, the doshutdown scriptlet from step 8 gets invoked on a power failure. This is turn invokes the shutdown.sh script on the ESXi host, which signals the VMWare Tools package in each VM on your host to do a clean shut down via the guest OS. In my experience, it works better than the PowerChute software from APC.

If you like to monitor things from your VMs, you can setup slave apcupsd instances on them that connect to the master UPS control Linux VM. Your slave apcupsd.conf files should have an entry like this:
UPSTYPE net < your UPS control VM IP >:3551
Entries like UPSCABLE and such do not matter in this case. This works with the Windows version of apcupsd (available here) as well. You can use the included apctray.exe to check out the current status of things.

That pretty much covers it, I think.

Solution 3:

According to APC, this is not possible and you require Powerchute Network shutdown. We tried this a number of times with USB and found no solution.

VMWare has info here on using the APC approved solution.

Would also think SmartUPS would be a better choice and you can fit with network card. Naturally more money but if your servers are at all important, that cost should be worth it. Also gives you more monitoring and alerting which might be useful at a remote site. You also need to assure sufficient runtime for all VMs to cleanly shutdown and then shutdown the host

Solution 4:

You might consider using the USB device passthrough functionality to a guest running PowerChute or other software able to monitor the UPS's health and capable of triggering a shutdown on the ESXi host (e.g. apcupsd). ESXi officially only supports a very limited number of USB devices for passthrough, but people have been attaching and passing through different classes of devices for a while already with varying success, but the APC UPS USB seems to work according to this walkthrough for a Windows VM or this one for a CentOS Linux VM.

Solution 5:

Have a look at vSphere Management Assistant (vMA) from here We use it at my office for doing what you are attempting, however with Smart-UPS connected via USB rather than Back-UPS.

Tags:

Electrical Power

Vmware Esxi

Apc Smart Ups

Recent Posts