Replicating messages from one Kafka topic to another kafka topic

While mirror makes works perfect for across the cluster solution, however, for same cluster your ducktap solution is not bad as MirrorMaker assumes you are pulling from one cluster to another cluster.

So a solution where you simply want to copy data between different topics in the same cluster, kafkacat is your friend.

export BOOTSTRAP_SERVERS=localhost:9096
export SOURCE_TOPIC=source_topic
export TARGET_TOPIC=target_topic

kafkacat -C -b $BOOTSTRAP_SERVERS -o beginning -e -t $SOURCE_TOPIC  | kafkacat -P -b $BOOTSTRAP_SERVERS  -t $TARGET_TOPIC

If you want to replicate data from one cluster to another then there is one kafka tool called MirrorMaker.

Kafka comes with a tool for mirroring data between Kafka clusters. The tool reads from a source cluster and writes to a destination cluster. Data will be read from topics in the source cluster and written to a topic with the same name in the destination cluster.

Here is syntax to run MirrorMaker tool:

bin/kafka-run-class.sh kafka.tools.MirrorMaker
       --consumer.config consumer.properties
       --producer.config producer.properties --whitelist my-topic

You can find this script in kafka installation directory. Here you need to provide consumer.properties of your source cluster and producer.properties of your destination cluster. You can whitelist which topics should be mirrored through --whitelist option.

You can find more information about Mirroring data between clusters

Note: MirrorMaker copies data into same topic_name in destination cluster as source cluster