Why are many mail servers using subdomains like mail.example.com?

The main reason to use a domain like mail.example.com or smtp.example.com to enable moving it without impacting other services running on the main domain. Only computers should be looking up the SMTP server, and that will be using the MX record for example.com Your users may need to provide the domains when setting up their client, but many (most?) will guess correctly based on their [email protected] email address.

With very few exceptions, only spambots use a second level (example.com), almost always spoofed, domain. The few legitimate servers using domains like example.com are usually broken in several other respects as well. Using a second level domain can contribute to the spam score for mail you are sending.

Professional administrators use dedicated sub-domains for mail services. In most cases, this will be a sub-domain that does not reflect the name of server on which the service is running.

I haven't had any problems sending or receiving SMTP mail on my server with a self-signed cert. The connecting site already knows it has the correct site based on your MX record. Recently, I have had a couple of servers fail on the startTLS command, but they reconnected without TLS. I believe they were using SSLv3 and got rejected on that basis, rather than a CA basis. A review of the most recent failures shows that spambots are disconnecting incorrectly generating errors. A significant portion of my email traffic is now TLS encrypted and there is no problems the certificate from my self-signed CA.

Using trusted certificates is one way to authenticate co-operating servers within a domain. However, you would want to either restrict the signed domains or use a private (self-signed) CA. There are other cases, such as a VPN, where you might also use a private CA.

Where you may run into issues is with Dovecot, but it will be your users who will be connecting. This is not a service for the general public to use. If you generated a CA to sign your certificate, you can make the CA certificate available to your users to import into their email client. This will resolve the issue.

You can get a signed certificate for free now. If not, the Let's Encrypt Certificate Authority should be available this fall. (It is now available.)