SQS dead letter queue not triggered on AWS Lambda invocation errors
I believe this is because DynamoDB streams are stream based event sources. The lambda documentation states that when dealing with stream based event sources "if a Lambda function fails, AWS Lambda attempts to process the erring batch of records until the time the data expires"
From my understanding, the lambda function will retry until the event is either processed successfully or expires and disappears from the stream, the event is never "discarded" by the lambda function, as they are in non-stream based event sources.
You may have to implement your own failure handling as a part of your main lambda function if you wish to discard certain events, posting the event manually to a queue/topic and returning succesfully.
- Using DynamoDB streams to trigger lambda means that you are using synchronous invocation. However, DLQ is only available for asynchronous invocations.
- The good news is that in November 2019, AWS published new error handling mechanisms for Kinesis and DynamoDB event source.
With this feature, you can configure a destination on failure. This destination can be an SNS topic, SQS queue, another lambda function, or an EventBridge event bus.
For adding this through the console UI,
- Go to the lambda function
- Click on the
Add Destination
button - Select Stream invocation
- Select on failure condition
- Select SQS queue as the destination and point it to the SQS that you want to use like a DLQ.
For adding it through cloudformation, follow this documentation.
I'll provide a basic example for the trigger that you need to attach to your lambda function:
LambdaTrigger:
Type: AWS::Lambda::EventSourceMapping
Properties:
FunctionName: !GetAtt Lambda.Arn
EventSourceArn: !GetAtt TableName.StreamArn
DestinationConfig:
OnFailure:
Destination: !GetAtt DLQ.Arn