Will using a self-signed SSL certificate on a mail server hinder communication?
You're fine with a self-signed certificate.
In my experience over several years, I never had any problems with a self-signed certificate on my mail server. I eventually switched to one signed by StartSSL and there was no discernible difference in interoperability.
See also this post on ServerFault.
The bottom line is, STARTTLS is about encryption, not authentication. While it can be locked down to perform authentication, the default of most (if not all) SMTP servers is opportunistic encryption. SMTP is plaintext by default, and needed encryption more than it needed authentication, thus the emphasis.
It is possible to lock down connections - for example, I've seen an Ironport configured to require a valid certificate signed by a known CA it liked (in this case, it liked Entrust but not Comodo, so we ended up changing the cert because it was easier than debugging somebody else's Ironport). But that's for locking down specific domains which want to ensure partnered encryption, I've never seen someone doing that with opportunistic encryption.
Encryption with SSL consists of the following steps:
- Identify the peer using a certificate.
- Create and exchange keys for the following communication.
- Exchange data, encrypted with the previously exchanged keys.
You are usually employing encryption because you think that anybody might listen to the connection. A small step from this passive listening is active manipulation of the connection which SSL can detect too. But for this it needs to properly identify your peer, otherwise anybody in the middle could claim to be the right server.
With a self-signed certificate proper identification is not possible, because anybody could create such a certificate with the same credentials as you did. This means that nobody should trust a self-signed certificate by itself. It should only trust a self-signed certificate if it could verify it using another way. This other way could be a manual configuration at the server side (usually not the case unless the sender knows the recipient directly) or the verification of the certificate through DANE, where the owner of the domain provides their certificate or fingerprint inside a trusted DNS record (needs DNSSec).
What servers actually do when they detect an untrusted certificate differs: some will accept the certificate and write verify=NO in the Received header of the mail while others might refuse to connect with SSL because they consider the connection insecure and downgrade to plain text or not connect at all. It depends on the settings of the sending server.
Currently probably most mail servers will successfully connect to other servers, even if they can't verify their certificate. But, more and more are using "real" (verifiable) certificates. If you compare the reports from Facebook from May 2014 to August 2014 you can see the use of verifiable certificates move from 30% up to 95% in just 4 month. This means, that the use of real certificates will probably required by more servers in the near future.
This means, that using a certificate signed by a well known CA is definitely better than using a self-signed certificate. But even that might not provide the security you are trying to achieve. Any of the 100th of CA trusted by the operating system can issue such a certificate, so an attacker might try to get a certificate for your domain too. But this is at least much harder then just create another self-signed certificate, because the attacker needs to either hack the CA or control your mail domain somehow to capture mails the CA use inside the identification process.
Another attack of an active man-in-the-middle man is to re-route the mail by attacking DNS: To route a mail from the sending mail server to the server responsible for your domain, the sending server needs to to a DNS lookup for the MX record of your domain, which contains the names of your mail servers. An active attacker near the sending server could attack this DNS lookup and provide a fake MX record which points to its own servers. TLS will not help here anymore. Instead you have to use DNSSec to secure your DNS records and the sender has to verify this record.
Apart from that TLS does not provide end-to-end security for mail, just hop-by-hop. Any mail server in between has access to the plain mail and thus needs to be fully trusted. To get end-to-end encryption you need to use S/MIME or PGP. And because of this some server admins don't care about proper TLS at all, because TLS will not offer real (end-to-end) security for the mails, even if properly implemented.