Amazon Aurora 1.8 Load Data From S3 - Cannot Instantiate S3 Client

I had the same issue. I tried adding AmazonS3FullAccess to the IAM role that my RDS instances were using...no joy.

After poking around, I went into the RDS console, to Clusters. Selected my Aurora cluster and clicked Manage IAM Roles. It gave me a drop-down, I selected the IAM role (same one that the individual instances were using).

Once I did that, all was well and data load was nice and fast.

So, there are (for us) 5 steps/components:

1) The S3 bucket and bucket policy to allow a user to upload the object

{
    "Version": "2012-10-17",
    "Id": "Policy1453918146601",
    "Statement": [
        {
            "Sid": "Stmt1453917898368",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account id>:<user/group/role>/<IAM User/Group/Role>"
            },
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket name>/*"
        }
    ]
}

The "Principal" would be whatever IAM user, group or role will be uploading the data files to the bucket so that the RDS instance can import the data.

2) The IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1486490368000",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name>/*"
            ]
        }
    ]
}

This is pretty simple with the Policy Generator.

3) Create the IAM Role:

This role should be assigned the IAM policy above. You can probably do an inline policy, too, if you're not going to use this policy for other roles down the line, but I like the idea of having a defined policy that I can reference later if I have a need.

4) Configure a Parameter Group that your cluster/instances will use to set the aws_default_s3_role value to the ARN of the role from #3 above.

5) Configure the Aurora Cluster by going to Clusters, selecting your cluster, selecting Manage IAM Roles and setting the IAM Role for your DB Cluster

At least for me, these steps worked like a charm.

Hope that helps!

If the only error is Internal error: Unable to initialize S3Stream and it throws this error immediately, possible culprits are:

• typo in the bucket or object name
• bucket created in different region than database
• bucket or object name is not specified according to the syntax for specifying a path to files stored on an Amazon S3 bucket: s3-region://bucket-name/file-name-or-prefix

The path includes the following values:

• region (optional) – The AWS Region that contains the Amazon S3 bucket to load from. This value is optional. If you don't specify a region value, then Aurora loads your file from Amazon S3 in the same region as your DB cluster.
• bucket-name – The name of the Amazon S3 bucket that contains the data to load. Object prefixes that identify a virtual folder path are supported.
• file-name-or-prefix – The name of the Amazon S3 text file or XML file, or a prefix that identifies one or more text or XML files to load. You can also specify a manifest file that identifies one or more text files to load.