Can SSL be used to encrypt non-HTTP data?
Can we use SSL/TLS to encrypt the protocol's payload without tunneling it through HTTP?
Absolutely. TLS provides secure communication on top of the transport layer and you can easily employ it as a transparent wrapper around your own custom protocols.
One advantage of TLS is that it is application protocol independent. Higher-level protocols can layer on top of the TLS protocol transparently. The TLS standard, however, does not specify how protocols add security with TLS; the decisions on how to initiate TLS handshaking and how to interpret the authentication certificates exchanged are left to the judgment of the designers and implementors of protocols that run on top of TLS.
(from RFC 5246 for TLS 1.2)
HTTP just happens to be one possible application-layer protocol that is commonly transmitted over TLS. There are many other examples where TLS is added to secure a protocol that has no built-in encryption. E.g., if you use a desktop email client, the communication with the mail server (probably using IMAP/POP3/SMTP) will likely be wrapped in TLS, too. TLS can also be used as an encrypted tunnel for the entire network stack for VPN applications (although OpenVPN only uses TLS for authentication, not for for encrypting the actual data - thanks, @ysdx).
Yes, TLS can be used for general transport layer security (as the name suggests). A few common uses:
- HTTP (HTTPS)
- FTP (FTPS)
- SMTP (SMTPS)
- VPN