Getting "SSL_connect returned=1 errno=0 state=error: certificate verify failed" when connecting to S3

With the investigative help of @RodrigoM and your question update, it all started to make sense. There are actually two distinct problems that contribute to the error you observe:

• Your openssl installation does not have the certificate chain needed to verify the Amazon server in its trusted certs store...
• ...which is the exact situation that should be solved by adding Aws.use_bundled_cert! to an initializer, according to the docs. But in this case it does not work because even though this command instructs the ruby openssl library to add various CA certs to the trusted store from the aws-sdk-core gem's CA bundle file, the file also does not contain the proper CA certificate as it is itself almost 2 years old and outdated. The intermediate CA cert CN=DigiCert Baltimore CA-2 G2 has been published Dec 8, 2015, so no wonder that the CA bundle file does not contain it.

Now, you have two options:

• You can try to install this intermediate CA certificate, probably including the root CA cert (CN=Baltimore CyberTrust Root), to your openssl trusted certs store. This should make the s_client command work. But you might still run into issues using these trusted certs from ruby code. For concrete steps for making it work under ruby on OSX, refer to the Solution section of this SO question.

• Also, since you are using a forked aws-sdk-ruby gem repository anyway, you may as well update the ca-bundle.crt file in your repo by adding the intermediate CA cert yourself (the root CA cert seems to be already present in the bundle). For this you need to do the following:

  • download the intermediate CA cert from the official page of the DigicertCA certificates (you can as well use the direct link above, but to obey security rules precisely you should also check the fingerprints)

  • convert it to the PEM format (it gets downloaded in DER format) and add it to the cert bundle using the following openssl command:

    openssl x509 -in DigiCertBaltimoreCA-2G2.crt -inform DER >> ca-bundle.crt
    

    after running this command, your ca-bundle.crt should contain the intermediate CA certificate at the end of the file.

  • Now simply push this updated bundle file to your repo and the Aws.use_bundled_cert! should start working!

  • If you care, perhaps the best would be also to start a github issue at the aws-sdk-ruby gem so that they update the cert bundle in their repo too...

Your Ruby code, AWS SDK etc are all good. This is not a Ruby or SDK issue. The error message you initially describe, and your later post of the OpenSSL connect log both point to the root cause of the problem: A missing root certificate and/or improperly configured CA cert bundle in the OpenSSL framework. A further clue is that the same code works on production. Its not the code.

The original error message itself points to OpenSSL certificate verification errors as you indicate. The stack traces also show Ruby internal lib errors in 2.3.1/lib/ruby/2.3.0/net/http.rb. This is core network library that leverages the OpenSSL framework.

The openssl s_client connect log more clearly shows the extact error number and message:

depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
verify error:num=20:unable to get local issuer certificate

Until you get a Verify return code: 0 (ok) on the DigiCert Baltimore CA-2 CA with the openssl s-client test, your code will not work.

The DigiCert Baltimore CA-2 CA cert is not present or not properly referenced by your OpenSSL setup on that machine. This is very common problem in broken or incomplete OpenSSL installations. You need to download that certificate, convert to PEM format, save it in a ca-certificate.crt file in your OpenSSL certs folder, then reference that file in your config, or in an environment variable SSL_CERT_FILE.

You can see a good solution for your root issue here

NOTE: To futher confirm this solution, you should run the openssl s_client test on your production server. You should see it verifying the same CA without problem. Check the OpenSSL config and CA cert bundle configuration there to see why there is a difference between your production and development environment.