Prevent DOS against RSA authentication

You could switch to a faster algorithm. RSA is fine, but Elliptic Curve Diffie-Hellman is faster. Let's take my current machine, a laptop with an AMD A8-4555M CPU (1.6 GHz, not a very fast processor):

$ openssl speed rsa2048 ecdhp256
Doing 2048 bit private rsa's for 10s: 1468 2048 bit private RSA's in 9.99s
Doing 2048 bit public rsa's for 10s: 47384 2048 bit public RSA's in 9.98s
Doing 256 bit  ecdh's for 10s: 5847 256-bit ECDH ops in 9.99s
OpenSSL 1.0.1c 10 May 2012
built on: Tue Mar 19 19:10:34 UTC 2013
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_NO_TLS1_2_CLIENT -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
                  sign    verify    sign/s verify/s
rsa 2048 bits 0.006805s 0.000211s    146.9   4747.9
                              op      op/s
 256 bit ecdh (nistp256)   0.0017s    585.3

So 256-bit ECDH, arguably stronger (or at least as strong) as 2048-bit RSA, is also about four times faster. This performance is for one core; since that specific CPU is quadri-core, it could handle two thousands connections per second. A server processor would probably do substantially better.

---

Among other possible mitigation techniques, you can implement some client-side delays. In your "accidental DoS" scenario, all the client try to reconnect simultaneously. Possibly, you could make the clients wait a random time before reconnecting, when the connection is lost (instead of connecting immediately, make them wait a random time between 0 and 60 seconds, then try again). This will help spread the load.

You could also buy more servers. Hardware is cheap.