RabbitMQ plugin to remove duplicate messages
Ok, i've read about RabbitMQ inner architecture and find out it's impossible. So the way around for somebody looking for it.
- Send only document
ID
in message body - Create a key-value store for worker (i use memcached for this). Key is
ID
value is timestamp of last worker run for thisID
. - When worker receives the message it checks if message timestamp greater then the one from key-value store. If it is, then update timestamp in the store and run the task, otherwise just skip it.
You can check this plugin I wrote which allows to de-duplicate messages published within the broker.
You can de-duplicate on the exchange or at the queue according to your needs. Only thing your publisher needs to do is to set the x-deduplicate-message
message header with the ID
of your message.