AWS SQS not receiving SNS messages
If the SQS is encryted then the event pushing messages to queue must follow below steps
Several AWS services send events to Amazon SQS queues. To allow these event sources to work with encrypted queues, you must perform the following steps.
Use a customer managed CMK.
To allow the AWS service to have the kms:GenerateDataKey* and kms:Decrypt permissions, add the following statement to the CMK policy.
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "service.amazonaws.com" }, "Action": [ "kms:GenerateDataKey*", "kms:Decrypt" ], "Resource": "*" }] }
Create a new SSE queue or configure an existing SSE queue using the ARN of your CMK.
Provide the ARN of the encrypted queue to the event source.
For sns replace service section with sns.amazonaws.com
"Principal": {
"Service": "sns.amazonaws.com"
}
In my case SQS wan't receiving messages from SNS because SQS had encryption turned ON. When I turned OFF encryption on SQS it started working!
This AWS documentation explains how to enable SNS compatibility with encrypted SQS queues:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html#compatibility-with-aws-services
SNS requires extra permissions to be able to use the KMS key to encrypt messages for the queue.
Thank you to Mark B for his answer. It provided the start to getting this working. However, in order to make a policy document work via the CLI there are a few quirks that aren't covered in the docs.
- There are all sorts of errors trying to pass json directly to the
--attributes
flag inaws sqs set-queue-attributes
command. For some reason it requires the modifying json to be in a.json
document referenced by the cli. - In the
.json
file provided to the cli, all of the double quotes inside the"Policy"
value (nested json) must be escaped (i.e.{ \"Statement\": \"HelloWorld\" }
). If this is not followed, it will validation errors. I ended up needing to use the ascii escape characters in order properly format the output (\x5C
). - The json file must be referenced by using
file://local-location
in the--attributes
flag. It throws errors if this is not followed.
See the following elements I used for reference:
load_sqs.sh:
SQS_POLICY=
sqs-policy()
{
#First param is the queue arn, second param is the topic arn
SQS_POLICY=`printf '{ "Policy": "{\x5C\"Version\x5C\":\x5C\"2012-10-17\x5C\",\x5C\"Statement\x5C\":[{\x5C\"Sid\x5C\":\x5C\"CloudformationLambdaSQSPolicy\x5C\",\x5C\"Effect\x5C\":\x5C\"Allow\x5C\",\x5C\"Principal\x5C\":\x5C\"*\x5C\",\x5C\"Action\x5C\":\x5C\"sqs:SendMessage\x5C\",\x5C\"Resource\x5C\":\x5C\"%s\x5C\",\x5C\"Condition\x5C\":{\x5C\"ArnEquals\x5C\":{\x5C\"aws:SourceArn\x5C\":\x5C\"%s\x5C\"}}}]}" }' "$1" "$2"`
`echo $SQS_POLICY > $PWD/sqs-policy.json`
}
#SNS parameters
SNS_NAME="${NAME}_SNS"
SQS_NAME="${NAME}_SQS"
#Create SNS topic to send cloudformation notifications to
SNS_ARN=`aws sns create-topic --name ${SNS_NAME} | jq -r '.TopicArn'`
#Create SQS to send SNS to (holding SNS messages for lambda -^ up)
SQS_URL=`aws sqs create-queue --queue-name ${SQS_NAME} | jq -r '.QueueUrl'`
SQS_ARN=`aws sqs get-queue-attributes --queue-url ${SQS_URL} --attribute-names QueueArn | jq -r '.Attributes .QueueArn'`
#Add necessary SQS <--> SNS permissions
sqs-policy ${SQS_ARN} ${SNS_ARN}
`aws sqs set-queue-attributes --queue-url ${SQS_URL} --attributes file://sqs-policy.json`
#subscribe the queue to the notifications
aws sns subscribe --topic-arn ${SNS_ARN} --protocol sqs --notification-endpoint ${SQS_ARN}
sqs-policy.json:
{ "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"CloudformationLambdaSQSPolicy\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"sqs:SendMessage\",\"Resource\":\"ResourceARN\",\"Condition\":{\"ArnEquals\":{\"aws:SourceArn\":\"SourceARN\"}}}]}" }
It doesn't look like you have given the SNS topic permission to publish to the SQS queue. Look at step 2 in this walkthrough. You'll need to add a policy like this to the SQS queue:
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"MySQSPolicy001",
"Effect":"Allow",
"Principal":"*",
"Action":"sqs:SendMessage",
"Resource":"arn:aws:sqs:us-east-1:123456789012:MyQueue",
"Condition":{
"ArnEquals":{
"aws:SourceArn":"arn:aws:sns:us-east-1:123456789012:MyTopic"
}
}
}
]
}
Replacing the ARNs with the ones for your topic and queue.