Consume from a Kafka Cluster through SSH Tunnel

Not as far as I know.

The trick I used when I needed to do something similar was:

1. setup a virtual interface for each Kafka broker
2. open a tunnel to each broker so that broker n is bound to virtual interface n
3. configure your /etc/hosts file so that the advertised hostname of broker n is resolved to the ip of the virtual interface n.

Es.

Kafka brokers:

• broker1 (advertised as broker1.mykafkacluster)
• broker2 (advertised as broker2.mykafkacluster)

Virtual interfaces:

• veth1 (192.168.1.1)
• veth2 (192.168.1.2)

Tunnels:

• broker1: ssh -L 192.168.1.1:9092:broker1.mykafkacluster:9092 jumphost
• broker2: ssh -L 192.168.1.2:9092:broker1.mykafkacluster:9092 jumphost

/etc/hosts:

• 192.168.1.1 broker1.mykafkacluster
• 192.168.1.2 broker2.mykafkacluster

If you configure your system like this you should be able reach all the brokers in your Kafka cluster.

Note: if you configured your Kafka brokers to advertise an ip address instead of a hostname the procedure can still work but you need to configure the virtual interfaces with the same ip address that the broker advertises.

Try sshuttle like this:

sshuttle -r user@host broker-1-ip:port broker-2-ip:port broker-3-ip:port

Of course, the list of broker depends on advertised listeners broker setting.

You don't actually have to add virtual interfaces to acces the brokers via SSH tunnel if they advertise a hostname. It's enough to add a hosts entry in /etc/hosts of your client and bind the tunnel to the added name.

Assuming broker.kafkacluster is the advertised.hostname of your broker:

/etc/hosts:
127.0.2.1 broker.kafkacluster

Tunnel:
ssh -L broker.kafkacluster:9092:broker.kafkacluster:9092 <brokerhostip/name>

Absolutely best solution for me was to use kafkatunnel (https://github.com/simple-machines/kafka-tunnel). Worked like a charm.