Difference between self-signed CA and self-signed certificate

Both options are valid, option 2 is simpler.

Option 1 (setting up your own CA) is preferable when you need multiple certificates. In a company you might set up your own CA and install that CA's certificate in the root keystore of all clients. Those clients will then accept all certificates signed by your CA.

Option 2 (self-signing a certificate without a CA) is easier. If you just need a single certificate, then this is sufficient. Install it in the keystores of your clients and you are done. But when you need a second certificate, you need to install that again on all clients.

Here is a link with further information: Creating Certificate Authorities and self-signed SSL certificates

First, about the distinction between key and certificate (regarding "CA key"), there are 3 pieces used when talking about public-key certificates (typically X.509): the public key, the private key and the certificate. The public key and the private key form a pair. You can sign and decrypt with the private key, you can verify (a signature) and encrypt with the public key. The public key is intended to be distributed, whereas the private key is meant to be kept private.

A public-key certificate is the combination between a public key and various pieces of information (mostly regarding the identity of the owner of the key pair, whoever controls the private key), this combination being signed using the private key of the issuer of the certificate. An X.509 certificate has a subject distinguished name and an issuer distinguished name. The issuer name is the subject name of the certificate of the entity issuing the certificate. Self-signed certificates are a special case where the issuer and the subject are the same. By signing the content of a certificate (i.e. issuing the certificate), the issuer asserts its content, in particular, the binding between the key, the identity (the subject) and the various attributes (which may indicate intent or scope of usage for the certificate).

On top of this, the PKIX specification defines an extension (part of a given certificate) which indicates whether a certificate may be used as a CA certificate, that is, whether it can be used as an issuer for another certificate.

From this, you build a chain of certificates between the end-entity certificate (which is the one you want to verify, for a user or a server) and a CA certificate you trust. There may be intermediate CA certificates (issued by other CA certificates) between the end-entity certificate of your service and the CA certificate you trust. You don't strictly need a root CA at the top (a self-signed CA certificate), but it's often the case (you may choose to trust an intermediate CA certificate directly if you wish).

For your use case, if you generate a self-signed certificate for a specific service, whether it has the CA flag (basic constraints extension) doesn't really matter. You would need it to be a CA certificate to be able to issue other certificates (if you want to build your own PKI). If the certificate you generate for this service is a CA certificate, it shouldn't do any harm. What matters more is the way you can configure your client to trust that certificate for this particular server (browsers should let you make an explicit exception quite easily for example). If the configuration mechanism follows a PKI model (without using specific exceptions), since there won't be a need to build a chain (with just one certificate), you should be able to import the certificate directly as part of the trust anchors of your client, whether it's a CA certificate or not (but this may depend on the configuration mechanism of the client).

You can openssl x509 -noout -text -in $YOUR_CERT to see the differences between files contents:

In your self-signed CA, you can see：

    X509v3 extensions:                                                          
        X509v3 Basic Constraints:
            CA:TRUE, pathlen:0

And in your self-signed certificate, it's:

    X509v3 extensions:                                                          
        X509v3 Basic Constraints:
            CA:FALSE

If you need more certificates (C), you need to create a self-signed CA (ssCA).
If you need a single certificate, you can just create a self-signed certificate (SSC).
To trust the single certificate (SSC), you need to install SSC into the root keystore on your client.
To trust many certificates at once, you need to create a self-signed CA (ssCA), then install ssCA into the root keystore on your client.