NoBrokersAvailable: NoBrokersAvailable-Kafka Error

You cannot create partitions within a consumer. Partitions are created when you create a topic. For example, using command line tool:

bin/kafka-topics.sh \
  --zookeeper localhost:2181 \
  --create --topic myNewTopic \
  --partitions 10 \
  --replication-factor 3

This creates a new topic "myNewTopic" with 10 partitions (numbered from 0 to 9) and replication factor 3. (see http://docs.confluent.io/3.0.0/kafka/post-deployment.html#admin-operations and https://kafka.apache.org/documentation.html#quickstart_createtopic)

Within your consumer, if you call assign(), it means you want to consume the corresponding partition and this partition must exist already.

The problem for me, was the firewall rule as I am running Kafka on Google Cloud.

It was working for me yesterday and I was scratching my head today for 1 hour thinking about why it doesn't work anymore .

As the public IP address of my local system changes every time I connect to a different LAN or WiFi, I had to allow my local system's public IP in the firewall rules. I would suggest using a connection with a fixed public IP or to check for this whenever you switch/change your connection.

These small changes in the configurations take too much to debug and fix them. Felt like wasted an hour for this.

I had the same error during kafka streaming. The code below resolved my error: We need to define the API version in KafkaProducer.

KafkaProducer(bootstrap_servers=['localhost:9092'],
              api_version=(0,11,5),
              value_serializer=lambda x: dumps(x).encode('utf-8'))