Cannot connect to Docker-Kafka-broker and Kafka to solve several important parameters of internal and external network isolation

1. Background

Like the previous big data environment, etc. are built through Docker, whether it is learning, testing, migration is undoubtedly the fastest way, especially when you want to use the same environment in multiple environments, https:// github.com/hulichao/docker-bigdata , for building ideas, you can refer to this warehouse. In general, just pay attention to doing more port mapping than virtual machines . Port mapping, that is, the network, is actually relatively redundant, so in When setting up the environment of Kafka, the following problems happened to be encountered, as shown belowconnection to node 0 colud not be establised. Broker may not be avaliable

Kafka cannot connect to the broker. It is fine on the virtual machine, but there is a problem with the local connection using idea.

2. Thinking

It was a blog written after the fact, so some of the screenshots of troubleshooting problems at that time were not posted one by one, but only briefly described the process and the final solution, and led to the solution of Kafka's isolation of internal and external networks.

2.1 First, troubleshoot various network problems

For example, if ping www.baidu.comyou use the virtual machine network, check the port usage and telnet linux121 9092find that there is no problem. In addition, I checked the virtual network card of Docker. You can refer to another article, https://blog.csdn.net/hu_lichao/article/details/109538567 . no problem

2.2 Then began to consider the problem of Kafka

I didn’t think about it in this direction, so I had to use googleit to input keywords kafka docker idea 连不上Broker, as shown below

Then refer to the first article to read the key information: Kafka的客户端（生产者与消费者统称）要连接 Broker 需要经过一层认证，不通过认证就无法连接！Then continue to read down, the client connection to Kafka requires two layers of authentication, one is advertised.listenersthe connection and port configured in the Broker on Zk. Note: The more pitted point here is , You must compare the content in PLAINTEXT as it is to correspond to the content in the Java program, that is, regardless of how you get the connection to kafka in Docker in the virtual machine, in short, bootstrap.serversthe content you want to send must PLAINTEXTbe consistent with that. You need to configure the hosts of linux121 to map to the actual ip and port.

java code

zk get broker information

kafka-like server.properties-like placement

3. Kafka's internal and external network parameters

According to the previous troubleshooting, there are two important parameters, listenersand advertise.listenerswhat are these two parameters for? Here is the official translation,

Used to specify the address and port of the current Broker to publish services. Cooperate with advertised.listeners to isolate internal and external networks.

Here are a few related parameters:

listener.security.protocol.map

Internal and external network isolation configuration: mapping configuration of listener name and security protocol. For example, you can isolate the internal and external networks, even if they all use SSL. listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSLNote that the name of each listener can only appear once in the map.inter.broker.listener.name

Used to configure the listener name used for communication between brokers. The name must be in the advertised.listenerslist. E.ginter.broker.listener.name=EXTERNAL listeners

Used to configure the list of URIs and listener names monitored by the broker. Use commas to separate multiple URIs and listener names. If the listener name does not represent a security protocol, it must be configured listener.security.protocol.map. Each listener must use a different network port. advertised.listenersThe address needs to be published to zookeeper for the client to use, if the address used by the client is different from the listeners configuration. The zookeeper can be found in.get /brokers/ids/<broker.id>

In an IaaS environment, the network interface of this entry must be different from the network interface bound to the broker. If this entry is not set, listenersthe configuration used. Unlike listeners, this entry cannot use 0.0.0.0network ports. advertised.listenersThe address must be configured or part of the configuration in listeners.

Note : The function of INTERNAL and EXTERNAL is equivalent to two aliases for the PLAINTEXT protocol. By default, there can only be one protocol with the same name in the map, so write separately

The following is a typical common configuration:

The above is written using wolai notes, you can use my invitation code DJED4WIto register, the quota is limited, come grab it!

4. Reference

• https://www.cnblogs.com/hellxz/p/why_cnnect_to_kafka_always_failure.html

Wu Xie, Xiao San Ye, a little rookie in the background, big data, and artificial intelligence. Please pay attention to more