SQS Lambda - retry logic?

According this blog:

https://www.lucidchart.com/blog/cloud/5-reasons-why-sqs-lambda-triggers-are-a-big-deal

Leverage existing retry logic and dead letter queues. If the Lambda function does not return success, the message will not be deleted from the queue and will reappear after the visibility timeout has expired.

Here is how I did it.

1. Create Normal Queues (Immediate Delivery), Q1
2. Create Delay Queues (5 mins delay), Q2
3. Create DLQ (After retries), DLQ1

(Q1/Q2) SQS Trigger --> Lambda L1 (if failed, delete on (Q1/Q2), drop it on Q2) --> On Failure DLQ

When messages arrive on Q1 it triggers Lambda L1 if success goes from there. If fails, drop it to Q2 (which is a delayed queue). Every message that arrives on Q2 will have a delay of 5 minutes.

If your initial message can have a delay of 5 mins, then you might not need two queues. One queue should be good. If the initial delay is not acceptable then you need two queues. One another reason to have two queues, you will always have a way for new messages that comes in the path.

If you have a code failure in handling Q1/Q2 aws infrastructure will retry immediately for 3 times before it sends it to DLQ1. If you handle the error in the code, then you can get the pipeline to work with the timings you mentioned.

SQS Delay Queues:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html

SQS Lambda Architecture:

https://nordcloud.com/amazon-sqs-as-a-lambda-event-source/

Hope it helps.

Re-tries and re-tries "timeout" can all be configured directly in the SQS queue.

When you create a queue, set up the following attributes:

The Default Visibility Timeout will be the time that the message will be hidden once it has been received by your application. If the message fails during the lambda run and an exception is thrown, lambda will not delete any of the messages in the batch and all of them will eventually re-appear in the queue.

If you only want to try 3 times, you must set the SQS re-drive policy (AKA Dead Letter Queue)

The re-drive policy will enable your queue to redirect messages to a Dead Letter Queue (DLQ) after the message has re-appeared in the queue N number of times, where N is a number between 1 and 1000.

It is essential to understand that lambda will continue to process a failed message (a message that generates an exception in the code) until:

1. It is processed without any errors (lambda deletes the message)
2. The Message Retention Period expires (SQS deletes the message)
3. It is sent to the DLQ set in the SQS queue re-drive policy (SQS "moves" the message to the DLQ)
4. You delete the message from the queue directly in your code (User deletes the message)

Lambda will not dispose of this bad message otherwise.

---

Important observations

Lambda will not deal with failed messages

Based on several experiments I ran to understand the behavior of the SQS integration (the documentation on re-tries can be ambiguous).

Lambda will not delete failed messages and will continue to re-try them. Even if you have a Lambda DLQ setup, failed messages will not be sent to the lambda DLQ. Lambda fully relies on the configuration of the SQS queue for this purpose as stated in the lambda DLQ documentation.

Recommendation:

• Always use a re-drive policy in your SQS queue.

Exceptions will fail a whole batch of messages

As I stated earlier if there is an exception in your code while processing a message, the whole batch of messages is re-tried, it doesn't matter if some of the messages were processed correctly. If for some reason a downstream service is failing you may end up with messages that were processed in the DLQ.

Recommendation:

• Manually delete messages that have been processed correctly
• Ensure that your lambda function can process the same message more than once

Lambda concurrency limits and SQS side effects

The blog post "Lambda Concurrency Limits and SQS Triggers Don’t Mix Well (Sometimes)" describes how, if your concurrency limit is set too low, lambda may cause batches of messages to be throttled and the received attempt to be incremented without ever being processed.

Recommendation:

The post and Amazon's recommendations are:

• Set the queue’s visibility timeout to at least 6 times the timeout that you configure on your function.
• The extra time allows for Lambda to retry if your function execution is throttled while your function is processing a previous batch.
• Set the maxReceiveCount on the queue’s re-drive policy to at least 5. This will help avoid sending messages to the dead-letter queue due to throttling.
• Configure the dead-letter to retain failed messages long enough so that you can move them back later to be reprocessed

Fairly simple (if you execute the Lambda in a Async way) and without the need to do any coding. First of all: if you code will throw an error, AWS Lambda will retry 3 more times to execute you code. In this case if the external API was not accessible, there is a big change that by the third time AWS retries – the API will work. Plus the delay between the re-tries is random-ish meaning, there a is a delay between the re-tries.

If the worst happens, and the external API is not yet up, you can take advantage of the dead-letter queue (DLQ) feature that each lambda have. Which will push to SQS a message saying what went wrong, so you can take additional actions. In this case, keep re-trying until you make it.

You can read more here: https://docs.aws.amazon.com/lambda/latest/dg/dlq.html