How to know when a set of RabbitMQ tasks are complete?
Use a "response" queue. I don't know any specifics about RabbitMQ, so this is general:
- Have your parent process send out requests and keep track of how many it sent
- Make the parent process also wait on a specific response queue (that the children know about)
- Whenever a child finishes something (or can't finish for some reason), send a message to the response queue
- Whenever
numSent == numResponded
, you're done
Something to keep in mind is a timeout -- What happens if a child process dies? You have to do slightly more work, but basically:
- With every sent message, include some sort of ID, and add that ID and the current time to a hash table.
- For every response, remove that ID from the hash table
- Periodically walk the hash table and remove anything that has timed out
This is called the Request Reply Pattern.
Based on Brendan's extremely helpful answer, which should be accepted, I knocked up this quick diagram which be helpful to some.
I have implemented a workflow where the workflow state machine is implemented as a series of queues. A worker receives a message on one queue, processes the work, and then publishes the same message onto another queue. Then another type of worker process picks up that message, etc.
In your case, it sounds like you need to implement one of the patterns from Enterprise Integration Patterns (that is a free online book) and have a simple worker that collects messages until a set of work is done, and then processes a single message to a queue representing the next step in the workflow.