RSA vs. DSA for SSH authentication keys

Go with RSA.

DSA is faster for signature generation but slower for validation, slower when encrypting but faster when decrypting and security can be considered equivalent compared to an RSA key of equal key length. That's the punch line, now some justification.

The security of the RSA algorithm is based on the fact that factorization of large integers is known to be "difficult", whereas DSA security is based on the discrete logarithm problem. Today the fastest known algorithm for factoring large integers is the General Number Field Sieve, also the fastest algorithm to solve the discrete logarithm problem in finite fields modulo a large prime p as specified for DSA.

Now, if the security can be deemed as equal, we would of course favour the algorithm that is faster. But again, there is no clear winner.

You may have a look at this study or, if you have OpenSSL installed on your machine, run openssl speed. You will see that DSA performs faster in generating a signature but much slower when verifying a signature of the same key length. Verification is generally what you want to be faster if you deal e.g. with a signed document. The signature is generated once - so it's fine if this takes a bit longer - but the document signature may be verified much more often by end users.

Both do support some form of encryption method, RSA out of the box and DSA using an El Gamal. DSA is generally faster in decryption but slower for encryption, with RSA it's the other way round. Again you want decryption to be faster here because one encrypted document might be decrypted many times.

In commercial terms, RSA is clearly the winner, commercial RSA certificates are much more widely deployed than DSA certificates.

But I saved the killer argument for the end: man ssh-keygen says that a DSA key has to be exactly 1024 bits long to be compliant with NIST's FIPS 186-2. So although in theory longer DSA keys are possible (FIPS 186-3 also explicitly allows them) you are still restricted to 1024 bits. And if you take the considerations of this [article], we are no longer secure with 1024 bits for either RSA or DSA.

So today, you are better off with an RSA 2048 or 4096 bit key.

Right now the question is a bit broader: RSA vs. DSA vs. ECDSA vs. Ed25519. So:

A presentation at BlackHat 2013 suggests that significant advances have been made in solving the problems on complexity of which the strength of DSA and some other algorithms is founded, so they can be mathematically broken very soon. Moreover, the attack may be possible (but harder) to extend to RSA as well.

The presentation suggests using elliptic curve cryptography instead. The ECC algorithms supported by OpenSSH are ECDSA and, since OpenSSH 6.5, Ed25519.

OpenSSH only supports NIST curves for ECDSA and according to this study those curves look really suspicious for NSA backdoors. And if NSA can already crack it, then it won't be as hard to crack for somebody else as a proper curve would be. Ed25519 is the same thing but with a better curve, so it's the safest bet against the underlying algorithm being mathematically broken.

Also, DSA and ECDSA have a nasty property: they require a parameter usually called k to be completely random, secret, and unique. In practice that means that if you connect to your server from a machine with a poor random number generator and e.g. the the same k happens to be used twice, an observer of the traffic can figure out your private key. (source: Wikipedia on DSA and ECDSA, also this).

The bottom line is:

• Never use DSA or ECDSA.
• Ed25519 is probably the strongest mathematically (and also the fastest), but not yet widely supported. As a bonus, it has stronger encryption (password-protection) of the private key by default than other key types.
• RSA is the best bet if you can't use Ed25519.

In SSH, on the client side, the choice between RSA and DSA does not matter much, because both offer similar security for the same key size (use 2048 bits and you will be happy).

Historically, version 1 of the SSH protocol supported only RSA keys. When version 2 was defined, RSA was still patented, so support of DSA was added, so that an opensource patent-free implementation could be made. RSA patent expired more than 10 years ago, so there is no worry now.

Theoretically, in some very specific situations, you can have a performance issue with one or the other: if the server is a very small machine (say, an i486), it will prefer clients with RSA keys, because verifying a RSA signature is less computationally expensive than verifying a DSA signature. Conversely, a DSA signature is shorter (typically 64 bytes vs 256) so if you are very short on bandwidth you would prefer DSA. Anyway, you will have a hard time detecting those effects, let alone find them important.

On the server, a DSA key is preferred, because then the key exchange will use a transient Diffie-Hellman key, which opens the road for "Perfect Forward Secrecy" (i.e. if a bad guy steals the server private key, he still cannot decrypt past connections that he would have recorded).