Digital Signatures: Encrypting the Hash vs Signing the Hash?

The answer given by zaitsman is a good explanation of the topics related to your questions and I think should be the accepted answer, but just to help tie it back to your specific question of why encrypting the hash doesn't give you the same result as signing the hash (the rsa.SignHash(hashOfData, "SHA256") in your code):

Signing a hash doesn't just encrypt the hash data -- it also encrypts the name (or some identifier) of the hashing algorithm used to generate the hash along with it. Without that, the receiver wouldn't know what algorithm to use when computing their own hash (of the message being sent) to compare with the one they just decrypted in order to verify the authenticity of the message (which, of course, is the whole point).

When you encrypted the hash yourself (with rsa.Encrypt(hashOfData, false) and rsa.Encrypt(hashOfData, true)), you only encrypted the hash data and not the combination of hash data and algorithm identifier ("SHA256" in your code). In other words, you encrypted different data, so you got different (encrypted) results.

The reason the return value of that SignHash call does match the value returned by rsa.SignData(data, sha256) is that the latter method does the same thing, except it does the hashing and hash signing as one operation, so you don't have to compute the hash as a separate step if you don't need it for any purpose other than signing it.

From RSACryptoServiceProvider.SignData Method on MSDN:

Computes the hash value of the specified data and signs it.

Also see: Why does SignHash need to know what hash algorithm was used?

You need to separate concerns, this will help you understand the terminology.

Any arbitrary blob of data can be hashed and/or encrypted in any combination. Hash means: use a cryptographic algorithm to generate a value that is irreversible (that is, simply by knowing algorithm and hash you are unable to reconstitute original data) and consistent (that is, given the same data and algorithm, the value of the hash produced is always the same).

Encrypt means: use a cryptographic algorithm to encipher data (altogether or in blocks) with a given key (a key can be symmetric or asymmetric).

Sign means: Hash the data and Encrypt the hash with a given key. Then, given the pair (for asymmetric) or same (for symmetric) key, a consumer can validate that:

1. hash is matching, that means the data has not been altered in transit
2. hash did come from the source that at least has the pair key (for asymmetric) or same key (for symmetric)