Kafka access inside and outside docker
What I needed to do was to declare the LISTENERS as both binding to all interfaces, and then advertise them differently - one to the docker network, one to the host:
services:
zookeeper:
image: confluentinc/cp-zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_SYNC_LIMIT: 2
kafka:
image: confluentinc/cp-kafka
ports:
- 9094:9094
depends_on:
- zookeeper
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
Your config of the listeners looks correct, the problem is with your Docker Compose:
ports:
- "9094:9092"
You’re mapping 9094
(‘Outside’) back to 9092
(‘Inside’) and thus when you connect you’re connecting to the ‘Inside’ listener. If you remove this line of config then your listener set up should work as intended.
For more info, see http://rmoff.net/2018/08/02/kafka-listeners-explained/
Agreed to the answer from @Robin Moffatt that outside port 9094
is mapped to container's 9092
.
In addition to removing that configuration from docker-compose.yml, the outside port 9094
should be mapped to container's 9094
.
ports:
- "9094:9094"
PS: I couldn't add this as a comment to previous answer, hence adding it as a new answer.