Is Kafka Stream StateStore global over all instances or just local?

This depends on your view on a state store.

  1. In Kafka Streams a state is shared and thus each instance holds part of the overall application state. For example, using DSL stateful operator use a local RocksDB instance to hold their shard of the state. Thus, with this regard the state is local.

  2. On the other hand, all changes to the state are written into a Kafka topic. This topic does not "live" on the application host but in the Kafka cluster and consists of multiple partition and can be replicated. In case of an error, this changelog topic is used to recreate the state of the failed instance in another still running instance. Thus, as the changelog is accessible by all application instances, it can be considered to be global, too.

Keep in mind, that the changelog is the truth of the application state and the local stores are basically caches of shards of the state.

Moreover, in the WordCount example, a record stream (the data stream) gets partitioned by words, such that the count of one word will be maintained by a single instance (and different instances maintain the counts for different words).

For an architectural overview, I recommend http://docs.confluent.io/current/streams/architecture.html

Also this blog post should be interesting http://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/


If worth mentioning that there is a GlobalKTable improvement proposal

GlobalKTable will be fully replicated once per KafkaStreams instance. That is, each KafkaStreams instance will consume all partitions of the corresponding topic.

From the Confluent Platform's mailing list, I've got this information

You could start prototyping using Kafka 0.10.2 (or trunk) branch...

0.10.2-rc0 already has GlobalKTable!

Here's the actual PR.

And the person that told me that was Matthias J. Sax ;)