What command shows all of the topics and offsets of partitions in Kafka?
Kafka ships with some tools you can use to accomplish this.
List topics:
# ./bin/kafka-topics.sh --list --zookeeper localhost:2181
test_topic_1
test_topic_2
...
List partitions and offsets:
# ./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group test_group --topic test_topic --zookeeper localhost:2181
Group Topic Pid Offset logSize Lag Owner
test_group test_topic 0 698020 698021 1 test_group-0
test_group test_topic 1 235699 235699 0 test_group-1
test_group test_topic 2 117189 117189 0 test_group-2
Update for 0.9 (and higher) consumer APIs
If you're using the new apis, there's a new tool you can use: kafka-consumer-groups.sh
.
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group count_errors --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
count_errors logs 2 2908278 2908278 0 consumer-1_/10.8.0.55
count_errors logs 3 2907501 2907501 0 consumer-1_/10.8.0.43
count_errors logs 4 2907541 2907541 0 consumer-1_/10.8.0.177
count_errors logs 1 2907499 2907499 0 consumer-1_/10.8.0.115
count_errors logs 0 2907469 2907469 0 consumer-1_/10.8.0.126
You might want to try kt. It's also quite faster than the bundled kafka-topics
.
This is the current most complete info description you can get out of a topic with kt:
kt topic -brokers localhost:9092 -filter my_topic_name -partitions -leaders -replicas
It also outputs as JSON, so you can pipe it to jq
for further flexibility.