What does "Rebalancing" mean in Apache Kafka context?

When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up.

the command for this is: rebalance.max.retries and is set to 4 by default.

also, it might be happening if the following is true:

ZooKeeper session timeout. If the consumer fails to send a heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.

Hope this helps!

Rebalance is the re-assignment of partition ownership among consumers within a given consumer group. Remember that every consumer in a consumer group is assigned one or more topic partitions exclusively.

A Rebalance happens when:

• a consumer JOINS the group
• a consumer SHUTS DOWN cleanly
• a consumer is considered DEAD by the group coordinator. This may happen after a crash or when the consumer is busy with a long-running processing, which means that no heartbeats has been sent in the meanwhile by the consumer to the group coordinator within the configured session interval
• new partitions are added

Being a group coordinator (one of the brokers in the cluster) and a group leader (the first consumer that joins a group) designated for a consumer group, Rebalance can be more or less described as follows:

• the leader receives a list of all consumers in the group from the group coordinator (this will include all consumers that sent a heartbeat recently and which are therefore considered alive) and is responsible for assigning a subset of partitions to each consumer.
• After deciding on the partition assignment (Kafka has a couple built-in partition assignment policies), the group leader sends the list of assignments to the group coordinator, which sends this information to all the consumers.

This applies to Kafka 0.9, but I'm quite sure for newer versions is still valid.