Redis as a message broker

I pass data between application using two native redis command: rpush and blpop . "blpop blocks the connection when there are no elements to pop from any of the given lists".

• Data are passed in json format, between application using list as queue.
• Application that want send data (act as publisher) make a rpush on a list
• Application that want receive data (act as subscriber) make a blpop on the same list

The code shuold be (in perl language)

---

Sender (we assume an hash pass)

#Encode hash in json format
my $json_text = encode_json \%$hash_ref;

#Connect to redis and send to list
my $r = Redis->new(server => "127.0.0.1:6379");
$r->rpush("shared_queue","$json_text");
$r->quit;

---

Receiver (into a infinite loop)

while (1) {
    my $r = Redis->new(server => "127.0.0.1:6379");
    my @elem =$r->blpop("shared_queue",0);

    #Decode hash element
    my $hash_ref=decode_json($elem\[1]);

    #make some stuff
}

I find this way very usefull for many reasons:

• The element are stored into list, so temporary disabling of receiver has no information loss. When recevier restart, can process all items into the list.
• High rate of sender can be handled with multiple instance of receiver.
• Multiple sender can send data on unique list. In ths case should be easily implmented a data collector
• Receiver process that act as daemon can be monitored with specific tools (e.g. pm2)

One idea is to push the data to a list (LPUSH) and trim it (LTRIM), so it doesn't grow forever if there are no consumers. On the other end, the consumer would grab items from that list and process them. You can also use keyspace notifications, and be alerted each time an item is added to that queue.

From Redis 5, there is new data type called "Streams" which is append-only datastructure. The Redis streams can be used as reliable message queue with both point to point and multicast communication using consumer group concept Redis_Streams_MQ