Force Linux servers to automatically try to get an IP address if they lose it

Did they revert to APIPA addressing? 169.254.0.0/16? You can disable that with the NOZEROCONF=Yes directive in /etc/sysconfig/network.

You should also extend your DHCP lease to be twice the longest reasonable outage interval of the DHCP server. Clients will generally wait half of the lease period before trying to renew; if they only check in once every couple of days, you have considerable leeway to fix the server or stand up a new one.

We run fully-reserved DHCP in production; it has not been a reliability problem. In fact, I contend that static IP addressing is one of the last manual operations that folks cling to, and it's largely an anachronism. I've not done a formal analysis, but in three years we've had one DHCP related issue, and that was an admin fat-finger which misfired a "release now" script. On the histogram of "Oops" causes, DHCP is way out on the long-tail.

I know everyone loves to slag on Windows, but IME when APIPA is disabled and DHCP enabled, Windows machines are downright tenacious about holding onto whatever their last DHCP assignment was. I've powered up machines after months of mothball, and wireshark shows them asking for the last DHCP-assigned address.

I use static IP-addresses for servers, DHCP for personal computers only.

Whilst I believe DHCP makes some effort to avoid changing a devices IP-address, this is not quaranteed - at the expiry of a lease the computer might be allocated a different IP-address by DHCP. I've always assumed this would be a bad thing to let happen to a server in the middle of a workday.

You can set PERSISTENT_DHCLIENT=yes in /etc/sysconfig/network-scripts/ifcfg-ethX. This effectively removes the -1 flag that is sent to dhclient by default on RedHat style systems. From man dhclient:

The -1 flag cause dhclient to try once to get a lease. If it fails, dhclient exits with exit code two.

However, in the case that you have a power outage and your servers come back online before your switches do, this does not work due to https://bugzilla.redhat.com/show_bug.cgi?id=234075. Basically that bug says that ifup first check to see if a link is present on the interface before attempting to run dhclient. No link -> no dhclient.

My solution is to add this to root's crontab:

*/5 * * * * ifconfig eth0 | grep -q "inet addr:" || ifup eth0 2>&1 >/dev/null

Also add this to /etc/dhcp/dhclient.conf for speedy recovery:

retry 10
timeout 10