OpenSSL PKCS#7 vs. S/MIME
PKCS#7 is an old standard from RSA Labs, published later on as an "informational RFC". Then, a new versions was produced, as an "Internet standard", i.e. with the seal of approval from the powers-that-be at IETF; a new name was invented for that: CMS. Newer versions were subsequently defined: RFC 3369, RFC 3852, RFC 5652. You can consider CMS and PKCS#7 to both designate the same standard, which has several versions.
CMS (PKCS#7) is a format for applying encryption, signatures and/or integrity checks on arbitrary binary messages (which can be large). It can be nested. It is the basis for some protocols such as time stamping. It is also frequently (ab)used as a container for X.509 certificates: the SignedData type includes a field for a set of signatures (and that set can be empty), and also a field to store arbitrary X.509 certificates which are considered as "potentially useful" for whoever processes the object.
CMS is nominally backward compatible: there are version fields at various places, and a library which understands a version of CMS should be able to process messages from older versions; moreover, such a library should also produce messages which are compatible with other versions, inasmuch as it is feasible given the message contents. See section 5.1 for an example: the "version" fields are set to the minimal values which still make sense given what else is in the structure.
S/MIME is a protocol for doing "secure emails". S/MIME uses CMS for the cryptographic parts; what S/MIME adds over CMS is the following:
Rules for encoding CMS objects (which are binary) into something which can travel unscathed in emails (which are text-based). These rules build on MIME, so basically Base64 with some headers.
Canonicalization rules for signed but not encrypted emails, with "detached signatures". When an email is just signed, and the recipient might not be S/MIME-aware, it is convenient to send the signature as a CMS
SignedDataobject which does not contain the email data itself; the signature is joined to the email as an attachment. It is then important to define exactly how the email contents are to be encoded into a sequence of bytes for the purpose of signature verification (a single mis-encoded bit would imply verification failure).Restrictions and usage practices on the actual CMS objects: how much nesting is allowed, what types may appear...
Some extra attributes so that a sender may publish, in his signed messages, what kinds of cryptographic algorithms it supports and similar metadata.
In a way, we can think of S/MIME as the glue between CMS and emails.
OpenSSL is a library which implements some protocols, including some versions of PKCS#7 and CMS and S/MIME. The library also comes with command-line tools which expose, as a command-line interface, some functionalities of the library. The tools won't support anything that the library does not implement (the contrary would be surprising, to say the least), but the converse is not true: the library may implement features to which the command-line tools give no easy access, or no access at all.
Generally speaking, the command-line tools are not very "serious". They are convenient for some manual operations or a few scripts, but for robust applications you should use the library directly (it is a C library, but bindings exist for many programming languages).
Among the command line tools are the subcommands pkcs7, cms and smime. Confusingly, the cms command is meant to be used for S/MIME, and supports S/MIME, and turns out to be usable as a "generic" CMS support tool only as a byproduct. The OpenSSL developers created the new command cms in order to support recent versions of S/MIME (and CMS) because they felt that they could not update the smime command without breaking interface compatibility: third-party scripts using openssl smime would have become invalid, if the parameters for that command has been adjusted to account for newer S/MIME versions. So they just created a new command, called cms.
Theoretically, you should use openssl cms for everything related to S/MIME; but since that command gives access to the OpenSSL support code for the "new" versions of S/MIME and CMS, it necessarily allows you to produce S/MIME messages which are perfectly valid but use features that older implementations of S/MIME might not support. This is summarily described in the man page. Extensive tests with other S/MIME implementations (e.g. Thunderbird, Outlook...) are needed to know exactly what you can do with S/MIME without making your emails unreadable by other people.
OpenSSL's smime app supports an older S/MIME format using PKCS#7 CMS, I'm guessing (partly based on the standards.txt distributed with OpenSSL) this is S/MIME Version 3.
The cms app in newer versions supports S/MIME v3.1 format messages, which use a newer CMS specification , so it's still S/MIME.
The pkcs7 app handles only PKCS#7 v1.5 data, not any of the newer CMS versions.
I think nomenclature is less than clear, one might reasonably expect the cms app to behave like the the pkcs#7 app.
The S/MIME Version 3.1 specification is in RFC 3851. The current version of S/MIME v3.2 is defined in RFC 5751, where it describes it itself:
S/MIME provides a consistent way to send and receive secure MIME data.
"PKCS#7 Cryptographic Message Syntax" is concerned with the the format of the cryptographic data (signatures, digests, encryption). "CMS" is the name for the current form of this. This section of the CMS specification (RFC 5652) details changes over previous versions including PKCS#7 v1.5. CMS describes itself as
The CMS describes an encapsulation syntax for data protection. It supports digital signatures and encryption. The syntax allows multiple encapsulations; one encapsulation envelope can be nested inside another.
One of the important changes in CMS relative to PKCS#7 v1.5 is that CMS is less tied to specific cryptographic algorithms or key management schemes (other than for interoperability requirements), so it is more extensible (see RFC 3370 for example).
The standalone cms app appeared in openssl-1.0.0 (according to CHANGES anyway). As far as I understand it, various parts of the CMS data structures have their own independent version details, there isn't a simple "1.0" versioning system, the RFC number is probably the best way to identify an instance. Again, trusting standards.txt, OpenSSL's CMS support seems to be the RFC336 instance.
If you need to create the most compatible S/MIME messages, the smime app is probably the best one to use. If you want to create or process newer versions, cms is better (there's probably also a better chance of successfully using newer/stronger encryption and digest algorithms with cms). The RFCs also allude to incompatibilities in S/MIME signatures across versions, and incompatibilities between CMS and PKCS#7.