AWS Elasticsearch Service IAM Role based Access Policy

When using IAM service with AWS, you must sign your requests. curl doesn't support signed requests (which consists of hashing the request and adding a parameter to the header of the request). You can use one of their SDK's that has the signing algorithm built in, and then submit that request.

See: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html#signing-requests

You can find the SDKs for popular languages here: http://aws.amazon.com/tools/

First, you said you can't login to an EC2 instance to curl the ES instance? You can't login? Or you can't curl it from EC2?

I have my Elasticsearch (Service) instance open to the world (with nothing on it) and am able to curl it just fine, without signing. I changed the access policy to test, but unfortunately it takes forever to come back up after changing it...

My policy looks like this:

{   "Version": "2012-10-17",   "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:843348267853:domain/myDomain"
    }   
   ] 
}

I realize this isn't exactly what you want, but start off with this (open to the world), curl from outside AWS and test it. Then restrict it, that way you're able to isolate the issues.

Also, I think you have an issue with the "Principal" in your access policy. You have your EC2 Role. I understand why you're doing that, but I think the Principal requires a USER, not a role.

See below:

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-access-policies

Principal

Specifies the AWS account or IAM user that is allowed or denied access to a resource. Specifying a wildcard (*) enables anonymous access to the domain, which is not recommended. If you do enable anonymous access, we strongly recommend that you add an IP-based condition to restrict which IP addresses can submit requests to the Amazon ES domain.

EDIT 1

To be clear, you added the AmazonESFullAccess policy to the my-ec2-role? If you're going to use IAM access policies, I don't think you can have a resource based policy attached to it (which is what you're doing).

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_compare-resource-policies.html

For some AWS services, you can grant cross-account access to your resources. To do this, you attach a policy directly to the resource that you want to share, instead of using a role as a proxy. The resource that you want to share must support resource-based policies. Unlike a user-based policy, a resource-based policy specifies who (in the form of a list of AWS account ID numbers) can access that resource.

Possibly try removing the access policy altogether?

Why you don't create a proxy with elastic ip and allow your proxy to access your ES? Basically exists three forms that you can limit access in your ES:

• Allow everyone
• White IP list
• Signing the access key and secret key provided by AWS.

I'm using two forms, in my php apps I prefer to use proxy behind the connection to ES and in my nodejs app I prefer to sign my requests using the http-aws-es node module. It's useful to create a proxy environment because my users needs to access the kibana interface to see some reports and it's possible because they have configured the proxy in their browsers =)

I must recommend to you close the access to your ES indexes, because it's pretty easy to delete them, curl -XDELETE https://your_es_address/index anyone can do it but you can say: "how the others users will get my ES address?" and I will answer you: "Security based in dimness isn't a real security"

My security access policy is basically something like it:

http://pastebin.com/EUKT1ekX