redis for logging

Redis is in memory datastore. Direct persistence of data to disk is possible with Save or BGSAVE command. Persistence (RDB/AOF) is a feature in addition to storage in-memory.

Requirement mentioned is to store logs to disk. Using any of message queues (like RabbitMQ) instead of in-memory datastore should make things simple. (logs will not eat-up memory)

Applications generating logs can publish them on queues and with separate consumers consuming log messages and writing them to disk.

How can I query for results within a datetime range, by a specific user, of a particular category?

Every block of log should be saved as a structure (example for C/C++) something like this:

   struct log{
     long datatime;
     string userId;
     string message;
     string category;
   };

Serialize this structure to string and store it in Redis as value. Keys for such values would be like: key = userId + DELIMITER + category + DELIMITER + datatime

You can have function which gets all the keys back and split them to get list of data for your specific keyword.

You need to keep in mind that Redis is an in-memory database (even if it can persist the data to disk). The data you put in Redis have to fit in memory.

The proposal in the article you mention is about using Redis as a distributed queuing system. Worker processes dequeue the items and write them to disk, so there are not that many items in Redis memory. This design has a flaw: if the worker processes cannot write the data fast enough to disk, Redis memory consumption will explode - so it has to be limited by configuration (Redis maxmemory parameter) or software (trim the queue at insert time, or empty the queue when it is full).

Now your proposal does not really work since all the data you write in Redis will be kept in memory (even if they are persisted to disk by Redis itself).

Another point is you cannot query Redis. Redis is not a relational database, it supports no ad-hoc query mechanism, only commands involving previously defined access paths. If you want to search data with different parameters, you have to anticipate all the possible searches and build the relevant data structures (set, sorted sets, etc ...) at insert time.

Another store (MongoDB, or a relational database) will probably be a much better fit for your use case.

You can store logs with following structure:

"logs:{category}:{userid}:{datetime}" = message

And then request it as following:

"logs:*:{userid}:{datetime}"

Or

"logs:{category}:*:{datetime}"