Is it possible to add partitions to an existing topic in Kafka 0.8.2

For anyone who wants solution for newer Kafka versions.Please follow this method.

Kafka's entire data retention and transfer policy depends on partitions so be careful about effects of increasing partitions. (Kafka's newer versions display warning regarding this) Try to avoid configuration in which one broker has too many leader partitions.

There is simple 3 stage approach to this.

Step 1: Increase the partitions in topics

./bin/kafka-topics.sh --zookeeper localhost:9092 --alter --topic testKafka_5 --partitions 6

Step 2: Create a partitioning json file for given topic

{ "version":1, "partitions":[ {"topic":"testKafka_5","partition":0,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":1,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":2,"replicas":[1,2,0]}, {"topic":"testKafka_5","partition":3,"replicas":[0,1,2]}, {"topic":"testKafka_5","partition":4,"replicas":[2,1,0]}, {"topic":"testKafka_5","partition":5,"replicas":[1,2,0]} ]}

Create file with newer partition and replicas. It's better to expand replicas to different brokers but they should be present within same cluster. Take latency into consideration for distant replicas. Transfer the given file to your Kafka.

Step 3: Reassign partitions and verify

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json  --execute

./bin/kafka-reassign-partitions.sh --zookeeper localhost:9092 --reassignment-json-file bin/increase-replication-factor.json --verify

You can check the effects of your change using --describe command.

Looks like you can use this script instead:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name 
   --partitions 40 

In the code it looks like they do same thing:

 AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)

kafka-topics.sh executes this piece of code as well as AddPartitionsCommand used by kafka-add-partition script.

However you have to be aware of re-partitioning when using key:

Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.