Run an arbitrary command when a service fails

There is an OnFailure= directive in section [Unit], documented in systemd.unit(5). It is defined as follows:

A space-separated list of one or more units that are activated when this unit enters the "failed" state.

(Also there is an OnFailureJobMode= directive in the same section which allows to set job mode for activating OnFailure= units.)

You can also use ExecStopPost to run a command directly instead of starting a unit.

I wasn't happy with the OnFailure setting so I kept looking and found ExecStopPost.

The the following real example, if the main task fails then systemd will run a git command.

[Unit]
Description=SRI Dispenser Server
ConditionPathExists=|/usr/bin/
After=sri-boot-dsp.service

[Service]
WorkingDirectory=/usr/share/sri/configurations/transmitter

User=root

# This is task to run when this service starts
ExecStart=/usr/bin/python -m sri.DispenserServer

# If any of the ExecStart tasks fail, then ExecStopPost will run
ExecStopPost=/bin/git checkout -- .

Restart=always
RestartSec=10
KillSignal=SIGKILL


[Install]
WantedBy=multi-user.target

https://www.freedesktop.org/software/systemd/man/systemd.service.html

ExecStopPost= Additional commands that are executed after the service is stopped. This includes cases where the commands configured in ExecStop= were used, where the service does not have any ExecStop= defined, or where the service exited unexpectedly. This argument takes multiple command lines, following the same scheme as described for ExecStart=. Use of these settings is optional. Specifier and environment variable substitution is supported. Note that – unlike ExecStop= – commands specified with this setting are invoked when a service failed to start up correctly and is shut down again.

It is recommended to use this setting for clean-up operations that shall be executed even when the service failed to start up correctly. Commands configured with this setting need to be able to operate even if the service failed starting up half-way and left incompletely initialized data around. As the service's processes have been terminated already when the commands specified with this setting are executed they should not attempt to communicate with them.

Note that all commands that are configured with this setting are invoked with the result code of the service, as well as the main process' exit code and status, set in the $SERVICE_RESULT, $EXIT_CODE and $EXIT_STATUS environment variables, see systemd.exec(5) for details.