Kafka Connect running out of heap space
Even I was facing the issue could not start my producer and consumer for a given topic. Also deleted all unnecessary log files and topics.Even though that's not related to the issue.
Changing the kafka-run-class.sh
did not work for me. I changed the below files
kafka-console-consumer.sh
kafka-console-producer.sh
and stopped getting OOM error. Both consumer and producer worked fine after this.
Increased the size to KAFKA_HEAP_OPTS="-Xmx1G"
was 512m earlier.
I found another cause of this issue this morning. I was seeing this same exception except that I'm not using SSL and my messages are very small. The issue in my case turned out to be a misconfigured bootstrap-servers
url. If you configure that URL to be a server and port that is open but incorrect, you can cause this same exception. The Kafka folks are aware of the general issue and are tracking it here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM
When you have Kafka problems with
java.lang.OutOfMemoryError: Java heap space
it doesn't necessarily mean that it's a memory problem. Several Kafka admin tools like kafka-topics.sh
will mask the true error with this when trying to connect to an SSL PORT. The true (masked) error is SSL handshake failed
!
See this issue: https://issues.apache.org/jira/browse/KAFKA-4090
The solution is to include a properties file in your command (for kafka-topics.sh
this would be --command-config
) and to absolutely include this line:
security.protocol=SSL
You can control the max and initial heap size by setting the KAFKA_HEAP_OPTS
environment variable.
The following example sets a starting size of 512 MB and a maximum size of 1 GB:
KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties connect-s3-sink.properties
When running a Kafka command such as connect-standalone
, the kafka-run-class
script is invoked, which sets a default heap size of 256 MB in the KAFKA_HEAP_OPTS
environment variable if it is not already set.