How do you backup Zookeeper?
There's a very nice tool called zk-shell that can do an enormous amount of things with Zookeeper. It has a mirror command that can copy an entire Zookeeper tree recursively to/from Zookeeper or local JSON file.
Source & documentation: https://github.com/rgs1/zk_shell
Installation on Centos 7:
yum install python2-pip
pip install zk_shell
Example to back up a zookeeper tree to a local JSON file /tmp/zookeeper-backup.json:
zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'
Netflix provided a solution for this called exhibitor. It's a "ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization."
Zookeeper writes a snapshot once it determines that it has enough transactions and every new snapshot completely supersedes older ones. So the latest snapshot + the transaction log from the time of the snapshot is enough to recover to current state. To make the calculations easier, you can simply backup the last 3 snapshots(in case of corruption of the latest snap) and the transaction logs from the timestamp corresponding to the earliest snapshot. The links below have some more details.
- http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_dataFileManagement
- http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance