Does openssl refuse self signed certificates without basic constraints?
The issue described in the question can be reproduced with a simpler setup. I've created two self signed certificates for testing which only differ in that one has a CA flag (ss-ca.pem) while the other does not (ss-noca.pem). With openssl verify one can check if the certificate can be verified against a specific CA path.
The self signed certificate with CA:true gets successfully verified against itself ('OK') although it stumbles over X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT (error 18) while verifying the chain:
$ openssl verify -CAfile ss-ca.pem ss-ca.pem
ss-ca.pem: CN = test CA
error 18 at 0 depth lookup:self signed certificate
OK
With the self signed certificate with CA:false the verification does not succeed (no 'OK') and it shows the error X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY (error 20):
$ openssl verify -CAfile ss-noca.pem ss-noca.pem
ss-noca.pem: CN = test no CA
error 20 at 0 depth lookup:unable to get local issuer certificate
My theory is that OpenSSL tries to build the trust chain to a certificate given with -CAfile. To build the trust chain the issuer certificate subject must match the issuer of the certificate, the signature must be valid (i.e. validated using the issuers public key) and the issuer certificate must be allowed to sign certificates, i.e. CA:true. While the first two checks returns no problems with both certificates the check for CA:true is only valid for ss-ca.pem.
I think the main difference from OpenSSL -CAfile to other concepts of a trust store is that -CAfile only does what it name implies: it contain a list of trusted CA which are used to validate the trust chain. Contrary to this other trust store implementations might also contain any kind of certificates where the validation routine simply checks if the certificate send by the server is directly trusted because it is contained in the trust store, no matter what kind of flags the certificate has or even if it is expired.
This difference between a general purpose trust store and OpenSSL -CAfile can also be seen when putting a non-CA end entity certificate as trusted into the store. In the example I use a CA certificate ca.pem which has issued a end entity certificate server.pem which of course does not have CA:true.
Validating the end entity certificate against the CA certificate works as expected:
$ openssl verify -CAfile ca.pem server.pem
server.pem: OK
But trying to trusting the end entity certificate directly by putting it into the CA store does not work because the CA store is not a general purpose trust store but limited to CA certificates:
$ openssl verify -CAfile server.pem server.pem
error 20 at 0 depth lookup:unable to get local issuer certificate
With a general purpose trust store the last verification should have succeeded too because the end entity certificate was explicitly declared as trusted.
This issue has been fixed today in OpenSSL (see https://github.com/openssl/openssl/issues/1418); should be available soon.