kafka问题解决:生产者消费者不能连接远程kafka集群

本文介绍kafka使用时连接超时的另一种错误:在单机上生产消费不会出问题,远程操作就会报错。由默认localhost引起

连接报错

生产者连接远程的kafka集群时,生产消息报错如下:

ERROR Error when sending message to topic first with key: null, value: 9 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

也有如下错误java api使用时报错连接localhost:9092失败

...:localhost:9092...
java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:51)
    at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:73)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:236)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:148)
    at java.lang.Thread.run(Thread.java:745)

分析处理

原因都是未能连接上kafka cluster。那么客户端已经指定了连接的ip和端口为什么还要去链接localhost,这个localhost是从哪来的呢?,首先api客户端或者生产者命令行都指定了正确的ip,那localhost就很有可能是kafka服务返回的,就类似servlet中request.getIp()的方法返回来的。

那么就可以先从kafka的启动配置文件中找找相关配置,因为这个数据一定是配置来的,否则如果实时获取到的应该是机器对应的真实外网ip。找到config/server.properties中的socket server settings配置部分:

############################# Socket Server Settings #############################


# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092


# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
# advertised.listeners=PLAINTEXT://your.host.name:9092

以上的注释中即有说明:broker的hostname和port用来给生产者和消费者广播。(实际上就是让生产者和消费者来连接broker的。)如果没有设置,就会使用listeners的配置,来作为java.net.InetAddress.getCanonicalHostName()的返回值。

到此基本可以知道,客户端(生产者消费者)是通过该方法java.net.InetAddress.getCanonicalHostName()的返回值作为连接broker的hostname和port的。

解决

添加一个host.name,配置advertised.listeners=PLAINTEXT://39.108.61.252:9092,注意你的kafka集群中都应该配置有效的advertised.listeners值,否则集群间不能通信

############################# Socket Server Settings #############################
host.name=39.108.61.252

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092


# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://39.108.61.252:9092

其它参考

来源stackOverFlow :Why can’t my consumers/producers connect to the brokers?

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what’s listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress. Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can’t be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the “hostname” property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.

其它参考

http://stackoverflow.com/questions/17808988/using-kafka-java-producer-send-a-message-producer-connection-to-localhost9092

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F

https://blog.csdn.net/henry_wu001/article/details/50245451

到此解决。若有说明不清晰或有误的,感谢提出

猜你喜欢

转载自blog.csdn.net/maoyuanming0806/article/details/80555979
今日推荐