Is it ok to use self-signed certificates for smtp transport?

Solution 1:

I ran an MTA with a self-signed certificate for a couple of years, until real ones got cheap enough that I could no longer be bothered to do so, and I didn't have a single rejection because of the unsigned certificate in all that time. I never had a single complaint about a mail being marked as spam because of it, either; if anything, using TLS often seems to mark you out as a non-spamming professional.

In my opinion, it is definitely worth enabling SMTP TLS if you can, whether or not you pay for a third-party-signed certificate.

Edit in response to your comment: It's not that someone couldn't choose to restrict inbound mail on that basis; I've never come across it, is all. A third-party-signed certificate is still useful to prove there is no man-in-the-middle attack happening; but that doesn't seem to be a serious problem in the MTA world at this time. If that starts to change, we could well find that people start to insist on signed certificates.

Security exists to address threats, so if the threat model changes, the range of sane and proportionate security responses will change with it.

Solution 2:

Like MadHatter said, at the moment encryption using a self-signed certificate is actually a step up in comparison to many other small time SMTPs if you are talking SMTP-relaying.

There are a few drawbacks, however, but not of the kind you expect. The most important one being your SMTP-clients choking on encryption. There are many small SMTP-clients deployed in off-the-shelf solutions for common business problems that don't like SMTP encryption and fail -- often silently! So, if you are using standard software with built-in mail clients, check them before you switch.

Other than that I'd say, if you are concernced about your mail being marked as spam, consider setting up SPF and DKIM. That helps very much.