Apache Kafka 分布式消息队列中间件安装与配置

其他 2018-07-17 05:15:29 阅读次数: 0
本文地址: http://blog.csdn.net/wangjia184/article/details/37921183

本文演示从1个zookeeper+1个kafka broker到3个zookeeper+2个kafka broker集群的配置过程。


kafka依赖于zookeeper, 首先下载zookeeper和kafka

$ wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
$ gzip -d zookeeper-3.4.6.tar.gz
$ tar -xvf zookeeper-3.4.6.tar

$ wget http://apache.fayea.com/apache-mirror/kafka/0.8.1.1/kafka_2.8.0-0.8.1.1.tgz
$ gtar xvzf kafka_2.8.0-0.8.1.1.tgz


对于CentOS来说在，在本地试验可能会遇到莫名其妙的问题，这一般是由于主机名不能正确识别导致。为了避免可能遇到的问题，首先查询本机主机名,

$ hostname
HOME
然后加入一条本地解析到/etc/hosts文件中 
127.0.0.1      HOME




一个zookeeper  + 一个kafka broker的配置

将zookeeper/conf/下的zoo_sample.cfg改名成zoo.cfg。 zookeeper默认会读取此配置文件。配置文件暂时不用改，默认即可

$mv zookeeper-3.4.6/conf/zoo_sample.cfg zookeeper-3.4.6/conf/zoo.cfg


启动Zookeeper服务, Zookeeper启动成功后在2181端口监听

$ zookeeper-3.4.6/bin/zkServer.sh start
JMX enabled by default
Using config: /home/wj/event/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED


启动Kafka服务，启动成功后在9092端口监听

$ kafka_2.8.0-0.8.1.1/bin/kafka-server-start.sh kafka_2.8.0-0.8.1.1/config/server.properties


扫描二维码关注公众号，回复： 2195474 查看本文章

开始测试

# 连接zookeeper, 创建一个名为test的topic, replication-factor 和 partitions 后面会解释，先设置为1
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

# 查看已经创建的topic列表
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test

# 查看此topic的属性
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

# 生产者连接Kafka Broker发布一个消息
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Hello World


消费者连接Zookeeper获取消息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Hello World

一个zookeeper  + 两个kafka broker的配置

为避免Kafka Broker的Singal-Point-Failure(单点失败), 需要建立多个Kafka Broker。先将kakfa目录中的/config/server.properties复制为/config/server-2.properties然后编辑它的内容， 具体见注释 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1  #每个Kafka Broker应该配置一个唯一的ID

############################# Socket Server Settings #############################

# The port the socket server listens on
port=19092   #因为是在同一台机器上开多个Broker，所以使用不同的端口号区分

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
#host.name=localhost   #如果有多个网卡地址，也可以将不同的Broker绑定到不同的网卡

############################# Log Basics #############################

# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2  #因为是在同一台机器上开多个Broker,需要确保使用不同的日志目录来避免冲突

现在就可以用新建的配置文件启动这个一个Broker了 
$ kafka_2.8.0-0.8.1.1/bin/kafka-server-start.sh kafka_2.8.0-0.8.1.1/config/server-2.properties


现在新创建一个topic, replication-factor表示该topic需要在不同的broker中保存几份，这里replication-factor设置为2, 表示在两个broker中保存。 

$ kafka_2.8.0-0.8.1.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test2

然后查看此topic的属性。

$ kafka_2.8.0-0.8.1.1/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2
Topic:test2	PartitionCount:1	ReplicationFactor:2	Configs:
	Topic: test2	Partition: 0	Leader: 0	Replicas: 0,1	Isr: 0,1
  • Leader: 如果有多个brokerBroker保存同一个topic，那么同时只能有一个Broker负责该topic的读写，其它的Broker作为实时备份。负责读写的Broker称为Leader.
  • Replicas : 表示该topic的0分区在0号和1号broker中保存
  • Isr : 表示当前有效的broker, Isr是Replicas的子集


在test2这个topic下发布新的消息验证工作正常

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test2
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
HHH
$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeeper lochost:2181 --from-beginning --topic test2
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
HHH

现在杀掉第一个Broker，模拟此点的崩溃 
$ ps aux | grep server.properties
user        2620  1.5  5.6 2082704 192424 pts/1  Sl+  08:57   0:25 java 
$ kill 2620

重新查询此topic的属性，会发现Leader已经进行了切换，而0号Broker也从Isr中消失了。 
$ kafka_2.8.0-0.8.1.1/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2
Topic:test2	PartitionCount:1	ReplicationFactor:2	Configs:
	Topic: test2	Partition: 0	Leader: 1	Replicas: 0,1	Isr: 1


$ kafka_2.8.0-0.8.1.1/bin/kafka-console-producer.sh --broker-list localhost:19092 --topic test2
# 使用1号broker再发布一个消息到test2下
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Another message

# 消费者查询，仍然工作正常
$ kafka_2.8.0-0.8.1.1/bin/kafka-consolconsumer.sh --zookeeper localhost:2181 --from-beginning92 --topic test2
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
HHH
Another message


三个zookeeper + 两个kafka broker的配置

同样，zookeeper也需要搭建cluster, 避免出现Single-Point-Failure. 由于zookeeper采用投票的方式来重新选举某节点失败后的leader, 所以至少需要三个zookeeper才能组成群集。且最好使用奇数个（而非偶数）。

下面是演示单机搭建最简单的zookeeper cluster, 具体的可以参考http://myjeeva.com/zookeeper-cluster-setup.html

#!/bin/sh

#下载
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

gzip -d zookeeper-3.4.6.tar.gz
tar xvf zookeeper-3.4.6.tar

#重命名 zoo_sample.cfg 为 zoo.cfg
mv zookeeper-3.4.6/conf/zoo_sample.cfg zookeeper-3.4.6/conf/zoo.cfg

#新建一个目录
sudo mkdir /usr/zookeeper-cluster
sudo chown -R jerry:jerry /usr/zookeeper-cluster

#3个子目录分别对应三个zookeeper服务
mkdir /usr/zookeeper-cluster/server1
mkdir /usr/zookeeper-cluster/server2
mkdir /usr/zookeeper-cluster/server3

#建立三个目录存放各自的数据文件
mkdir /usr/zookeeper-cluster/data
mkdir /usr/zookeeper-cluster/data/server1
mkdir /usr/zookeeper-cluster/data/server2
mkdir /usr/zookeeper-cluster/data/server3

#建立三个目录存放各自的日志文件
mkdir /usr/zookeeper-cluster/log
mkdir /usr/zookeeper-cluster/log/server1
mkdir /usr/zookeeper-cluster/log/server2
mkdir /usr/zookeeper-cluster/log/server3

#在每一个数据文件目录中，新建一个myid文件，文件必须是唯一的服务标识，在后面的配置中会用到
echo '1' > /usr/zookeeper-cluster/data/server1/myid
echo '2' > /usr/zookeeper-cluster/data/server2/myid
echo '3' > /usr/zookeeper-cluster/data/server3/myid

#将zookeeper复制三份
cp -rf zookeeper-3.4.6/* /usr/zookeeper-cluster/server1
cp -rf zookeeper-3.4.6/* /usr/zookeeper-cluster/server2
cp -rf zookeeper-3.4.6/* /usr/zookeeper-cluster/server3

然后编辑每个zookeeper的zoo.cfg配置文件。
将dataDir和dataLogDir设置为各自独立的目录；然后保证clientPort不会和其它zookeeper冲突（因为这里演示是3个实例安装于一台服务器上)

最后加入下面几行

server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888
server. X= IP: port1: port2

X是在该zookeeper数据文件目录中myid指定的服务ID.

IP是当前zookeeper绑定的IP地址,因为是演示，所以全都是localhost

port1 是Quorum Port

port2 是Leader Election Port

由于3个zookeeper在同一台机器上，需要使用不同的端口号避免冲突。

修改后的结果如下

/usr/zookeeper-cluster/server1/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/zookeeper-cluster/data/server1
dataLogDir=/usr/zookeeper-cluster/log/server1
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888

 /usr/zookeeper-cluster/server2/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/zookeeper-cluster/data/server2
dataLogDir=/usr/zookeeper-cluster/log/server2
# the port at which the clients will connect
clientPort=12181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888

  /usr/zookeeper-cluster/server3/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/zookeeper-cluster/data/server3
dataLogDir=/usr/zookeeper-cluster/log/server3

# the port at which the clients will connect
clientPort=22181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=0.0.0.0:2888:3888
server.2=0.0.0.0:12888:13888
server.3=0.0.0.0:22888:23888

然后分别启动3个zookeeper服务

$ /usr/zookeeper-cluster/server1/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/zookeeper-cluster/server1/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
$ /usr/zookeeper-cluster/server2/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/zookeeper-cluster/server2/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
$ /usr/zookeeper-cluster/server3/bin/zkServer.sh start
JMX enabled by default
Using config: /usr/zookeeper-cluster/server3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

启动完成后查看每个服务的状态，下面可以看到server2被选为了leader. 而其它2个服务为follower.

$ /usr/zookeeper-cluster/server1/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server1/bin/../conf/zoo.cfg
Mode: follower
$ /usr/zookeeper-cluster/server2/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server2/bin/../conf/zoo.cfg
Mode: leader
$ /usr/zookeeper-cluster/server3/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server3/bin/../conf/zoo.cfg
Mode: follower


接下来修改kafka的server.properties配置文件

kafka_2.8.0-0.8.1.1/config/server.properties和kafka_2.8.0-0.8.1.1/config/server-2.properties

将3个zookeeper的地址加入到zookeeper.connect中，如下:

zookeeper.connect=localhost:2181,localhost:12181,localhost:22181

启动2个Kafka broker

$ kafka_2.8.0-0.8.1.1/bin/kafka-server-start.sh kafka_2.8.0-0.8.1.1/config/server.properties
$ kafka_2.8.0-0.8.1.1/bin/kafka-server-start.sh kafka_2.8.0-0.8.1.1/config/server-2.properties

接下来验证一下

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test3 --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
fhsjdfhdsa
fjdsljfdsadsfdas

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeepecalhost:12181 --topic test3 --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
fhsjdfhdsa
fjdsljfdsadsfdas

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeepecalhost:22181 --topic test3 --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
fhsjdfhdsa
fjdsljfdsadsfdas

现在模拟leader挂掉的情况，直接将server2 的zookeeper杀掉

$ ps aux | grep server2
user     2493  1.0  1.8 1661116 53792 pts/0   Sl   14:46   0:02 java
$ kill 2493

重新查询一次各zookeeper的状态，会发现leader发生了改变

$ /usr/zookeeper-cluster/server3/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server3/bin/../conf/zoo.cfg
Mode: leader
$ /usr/zookeeper-cluster/server1/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server1/bin/../conf/zoo.cfg
Mode: follower
$ /usr/zookeeper-cluster/server2/bin/zkServer.sh status
JMX enabled by default
Using config: /usr/zookeeper-cluster/server2/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

再次验证，kafka集群仍然工作正常。

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test3 --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
fhsjdfhdsa
fjdsljfdsadsfdas

$ kafka_2.8.0-0.8.1.1/bin/kafka-console-consumer.sh --zookeepecalhost:22181 --topic test3 --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
fhsjdfhdsa
fjdsljfdsadsfdas

$ kafka_2.8.0-0.8.1.1/bin/katopics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test5
Created topic "test5".

$ kafka_2.8.0-0.8.1.1/bin/katopics.sh --create --zookeeper localhost:22181 --replication-factor 2 --partitions 2 --topic test6
Created topic "test6".
本文地址: http://blog.csdn.net/wangjia184/article/details/37921183

猜你喜欢

转载自blog.csdn.net/wangjia184/article/details/37921183
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
周排行
让自己的头脑极度开放
CentOS 6.5(x64) 和Redhat6.5操作系误删libc
高可用注册中心
【日记】12.28/【题解】AtCoder AGC041
XML（5）_XML 约束_DTD
Java集合Map（四）
树梅派安装桌面环境教程
pipenv 的 使用和安装
小程序白屏问题和内存研究
C语言简单选择排序
每日归档
更多
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022