SQS maxNumberOfMessages
Given there is a cost per request is the weighted random distribution based on amazon profit =))
It's clear that your goal here is to reduce costs, whether that be by sending fewer requests to SQS, or by forcing SQS to deliver the maximum amount of messages available.
As you've stated in your question, SQS has no obligation to deliver the maximum amount of messages available. However, there is something I'd like to inform you about, assuming you're not already aware of it.
Long Polling
The Developer Guide of Amazon's Simple Queue Service states:
The process of consuming messages from a queue depends on whether you use short or long polling. By default, Amazon SQS uses short polling, querying only a subset of its servers (based on a weighted random distribution) to determine whether any messages are available for a response. You can use long polling to reduce your costs while allowing your consumers to receive messages as soon as they arrive in the queue.
The messages that you sent to SQS may have all been stored on separate servers. As the documentation states, only a subset of servers may be queried if your queue is set to use short polling. My guess is that you were unlucky when invoking receiveMessage
and only 3
were returned every time.
If we look at the benefits of long polling on the same documentation page, it states:
Long polling offers the following benefits:
Eliminate empty responses by allowing Amazon SQS to wait until a message is available in a queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action.
Eliminate false empty responses by querying all—rather than a subset of—Amazon SQS servers.
The second bullet is very important here. Even though you're not seeing empty responses, there may exist more messages stored on servers that aren't being queried. If you enable long polling, you hopefully should see an increase in the amount of messages returned, assuming there's more than 3 servers in total.
Therefore, my suggestion is to enable long polling on your queue. To do this, see the Setting Up Long Polling page.
As DevilCode mentioned in his comment below, he was able to resolve his issue by using a FIFO queue instead of a standard queue, and by enabling long polling on it.