Is it possible to ensure unique messages are in a rabbitmq queue?

The core problem seems to be this:

"...its possible that a piece of data is broken down into a part that's 
duplicated in the queue and the consumers continue to process it and 
end up in a infinite loop."

You can focus on uniqueness of your queued items all you want, but the issue above is where you should focus your efforts, IMO. One way to prevent infinite looping might be to have a "visited" bit in your message payload that is set by consumers before they re-queue the broken-down item.

Another option would be to have the consumers re-queue back to a special queue that is treated slightly differently to prevent infinite looping. Either way, you should attack the issue by dealing with it as a core part of your application's strategy rather than using a feature of a messaging system to step around it.

I think even if you could fix the issue of not sending duplicates to the queue, you will sooner or later hit this issue:

From RabbitMQ Documentation: "Recovery from failure: in the event that a client is disconnected from the broker owing to failure of the node to which the client was connected, if the client was a publishing client, it's possible for the broker to have accepted and passed on messages from the client without the client having received confirmation for them; and likewise on the consuming side it's possible for the client to have issued acknowledgements for messages and have no idea whether or not those acknowledgements made it to the broker and were processed before the failure occurred. In short, you still need to make sure your consuming clients can identify and deal with duplicate messages."

Basically, it looks like this, you send a request to rabbitmq, rabbitmq replies with an ACK but for 1 reason or another, your consumer or producer does not receive this ACK. Rabbitmq has no way of knowing the ack was not received and your producer will end up re-sending the message, having never received an ack.

It is a pain to handle duplicate messages especially in apps where messaging is used as a kind of RPC, but it looks like this is unavoidable when using this kind of messaging architecture.