How to automatically restart a service on failure in Linux
Solution 1:
Only answering the service restart part. I came across Monit as well, but on CentOS 7 systemd takes care of all that for you. You just need to add these two lines to the .service file (if they're not there already):
Restart=always
RestartSec=3
See https://jonarcher.info/2015/08/ensure-systemd-services-restart-on-failure/ for reference.
If you want to create a custom systemd service, it's pretty straightforward to write your own service file. See the example below, for a custom http server.
Start the editor with a new service file:
vim /etc/systemd/system/httpd.service
And add the following content, which you can edit as required:
[Unit]
Description=My httpd Service
After=network.target
[Service]
Type=simple
User=root
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PERLLIB=/perl
ExecStart=/bin/httpd /etc/httpd.conf
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
I want it to start automatically on boot:
systemctl enable httpd
Tell systemd about the changes and start the service:
systemctl daemon-reload
systemctl start httpd
And now you can see the status:
systemctl status httpd
For reference, see https://scottlinux.com/2014/12/08/how-to-create-a-systemd-service-in-linux-centos-7/
Solution 2:
monit is a great way to monitor and restart services when they fail--and you'll probably end up using this for other essential services (such as Apache). There's a nice article on nixCraft detailing how to use this for services specifically, although monit itself has many more functions beyond this.
As for the socket aspect, @galraen answered this spot on.
Solution 3:
You can call setsockopt(2) for listening socket with SO_REUSEADDR, so you will be able to bind(2) it again without waiting for expiring all connections. Another possibility: drop connections from kernel. FreeBSD have tcpdrop command for this, don't know about Linux.
Solution 4:
If your linux distro uses Upstart instead of SysV init, then the respawn keyword does this.
http://upstart.ubuntu.com/cookbook/#respawn