zookeeper + KAFKA Cluster Setup
1600174884 ZooKeeper is a distributed, open-source coordination service for distributed applications, is an open source implementation of Google's Chubby, is a key component of Hadoop and Hbase. It is to provide a consistent service for distributed applications, provides features include: configuration maintenance, domain name service, distributed synchronization, cluster management. Because Kafka cluster is the state of information stored in Zookeeper, and the dynamic expansion of Kafka is achieved by Zookeeper, so it is necessary to build Zookeerper priority clusters, build distributed state management. We began to prepare the environment, to build a cluster:
So zookeeper is based on Java development environment then need to install the Java installation package used here zookeeper version zookeeper-3.4.14, the installation package version Kafka was kafka_2.11-2.2.0.
AMQP protocol: Advanced Message Queuing Protocol (Advanced Message Queuing Protocol) is an open standard messaging middleware application layer protocol. AMQP defines the format of the byte stream data transmitted through a network. Therefore, compatibility is very good, any program can achieve AMQP protocol and other programs compatible with the AMQP protocol interaction, it can be done easily cross-language, cross-platform.
server1：192.168.42.128
server2：192.168.42.129
server3：192.168.42.130
First check before installing the system does not come with open-jdk
command:
rpm -qa | grep java
rpm -qa | grep jdk
If there is no input information is not installed.
Retrieve a list of 1.8
yum list java-1.8 *
All files are installed 1.8.0
yum install java-1.8.0-openjdk * -y
Check whether the installation was successful by using the command
java -version
cat /etc/hosts
192.168.42.128 kafka01
192.168.42.129 kafka02
192.168.42.130 kafka03
1. Turn off selinux, firewall.
setenforce 0
systemctl stop firewalld && systemctl disable firewalld
2. Create a directory of zookeeper and Kafka:
cd /usr/local/
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.0/kafka_2.11-2.2.0.tgz
tar xzvf zookeeper-3.4.14.tar.gz
xzvf tar kafka211220tgz
mkdir -pv zookeeper / {zkdata, zkdatalog} #zkdata snapshot log is located, zkdatalog transaction log is located
mkdir -pv kafka / kfdatalogs #kfdatalogs message log is located
4. Build and change the zookeeper profile (on the need to set up three servers):
# Zoo_sample can be understood as a zookeeper comes profile template, a copy of a file ending .cfg configuration.
cp -of /usr/local/zookeeper/conf/{zoo_sample,zoo.cfg}
tickTime = heartbeat time between 2000 #zookeeper server.
initLimit = 10 #zookeeper maximum connection time to failure
syncLimit = synchronous communication time of 5 #zookeeper
dataDir = / usr / local / zookeeper / absolute path to store a snapshot of the log zkdata #zookeeper
Absolute path to store transaction log of dataLogDir = / usr / local / zookeeper / zkdatalog #zookeeper of
clientPort = 2181 #zookeeper connection port and the client
server.1 = 192.168.11.139: 2888: 3888 # server and its number, server IP address, communication port, the port election
server.2 = 192.168.11.140: 2888: 3888 # server and its number, server IP address, communication port, the port election
server.3 = 192.168.11.141: 2888: 3888 # server and its number, server IP address, communication port, the port election
# The above ports are zookeeper default port, can be modified with demand
5. Create myid file:
On server1
echo “1” > /usr/local/zookeeper/zkdata/myid
# Is on a different server, the server number to send myid under zkdata.
6. Start zookeeper cluster
cd /usr/local/zookeeper/bin
./zkServer.sh start
./zkServer.sh status
#Mode: leader master node, Mode: follower from the node, usually only one cluster ZK Leader, a plurality of follower, the main response is generally read and write requests of the client, and synchronization data from the master, when the master will hang from the follower a leader in the vote out.
zookeeper electoral process / Works
https://blog.csdn.net/wyqwilliam/article/details/83537139
In zookeeper cluster, each node there are the following three kinds of roles and four states:
Role: leader, follower, observer
状态：leading，following，observing，looking
Each Server There are four states in the course of their work:
LOOKING: Current Server does not know who is the leader, we are searching for.
LEADING: Current Server is the elected leader.
FOLLOWING: leader has been elected, the current Server to synchronize.
OBSERVING: observer act exactly the same in most cases with the follower, but they do not participate in the elections and vote, but only to accept the results (observing) elections and voting
This, zookeeper cluster build end, based on the following zookeeper build kafka cluster:
Kafka's basic concepts:
Topic: Topic especially Kafka message source process (feeds of messages) different classification.
Partition: the packet on the physical Partition Topic, a topic can be divided into a plurality of partition, each partition is an ordered queue. partition each message is assigned a sequential id (offset).
Message: news, is the basic unit of communication, each producer can publish some messages to a topic (theme).
Producers: data producers message, called the producers to process a topic Kafka published message.
Consumers: Consumers of data messages, subscribe to topics and process messages issued by its process is called consumers.
Broker: caching proxy, Kafka cluster of one or more servers collectively referred to as broker, here is the AMQP protocol.
cd /usr/local/kafka/config/
1. Modify server.properties file:
broker.id = 1 # myid here and in the same file zookeeper, using the unique identification
prot = 9092 #Kafka between ports connected to the cluster, not in the profile, but the default is 9092, can be modified with the demand, here we add
log.dirs = / usr / local / kafka / kfdatalogs # Kafka message log stored absolute path
advertised.listeners=PLAINTEXT://kafka01:9092
log.retention.hours = 168 # default maximum message persistence time of 168 hours, 7 days
message.max.byte = 5242880 # saved messages maximum 5M
default.replication.factor = the number of copies saved messages 2 #kafka, if one copy fails, the other can continue to provide services
replica.fetch.max.bytes = 5242880 # take the maximum number of messages directly
zookeeper.connect = 192.168.42.128: 2181,192.168.42.129: 2181,192.168.42.130: IP address and zookeeper 2181 # various nodes of the cluster of ports, port zookeeper cluster setup is the number of port here is how much.
Unmodified profile information:
The number of threads num.network.threads = 3 # This is borker network processing
num.io.threads = 8 # borker This is the I / O processing threads
num.partitions = 1 # default number of partitions, a topic a default partition number
log.retention.hours = 168 # default maximum message persistence time of 168 hours, 7 days
message.max.byte = 5242880 # saved messages maximum 5M
default.replication.factor = the number of copies saved messages 2 #kafka, if one copy fails, the other can continue to provide services
replica.fetch.max.bytes = 5242880 # take the maximum number of messages directly
log.segment.bytes = 1073741824 # The parameters are: kafka because the messages are in the form of an additional floor to the file, when more than this value, kafka will play a new file
log.retention.check.interval.ms = 300000 # 300,000 every millisecond to check the configuration of the above log time to failure (log.retention.hours = 168), to the directory to see if there is expired messages if there are, delete
log.cleaner.enable = false # whether to enable log compression, generally do not enable, enabled, then you can improve performance
2. Start Kafka cluster:
cd /usr/local/kafka/bin
bash kafka-server-start.sh -daemon …/config/server.properties
3. Starting test:
3.1 Creating topic
./kafka-topics.sh --create --zookeeper 192.168.21.241：2181 --replication-factor 2 --partitions 1 --topic wg01
# -Replication-factor 2 two copies
# -Partitions 1 to create a partition
# -Topic tian tian theme
3.2 Creating a producter:
./kafka-console-producer.sh --broker-list 192.168.21.241:9092 --topic wg01
3.3 Creating a consumer:
./kafka-console-consumer.sh --bootstrap-server 192.168.11.141:9092 --topic wg01 --from-beginning
3.4 View topic:
--list --zookeeper 192.168.11.141:2181 ./kafka-topics.sh
----------------
Disclaimer: This article is original CSDN bloggers "A large abandoned ❀" of article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/TH_lsq/article/details/102626967
1600174884 ZooKeeper is a distributed, open-source coordination service for distributed applications, is an open source implementation of Google's Chubby, is a key component of Hadoop and Hbase. It is to provide a consistent service for distributed applications, provides features include: configuration maintenance, domain name service, distributed synchronization, cluster management. Because Kafka cluster is the state of information stored in Zookeeper, and the dynamic expansion of Kafka is achieved by Zookeeper, so it is necessary to build Zookeerper priority clusters, build distributed state management. We began to prepare the environment, to build a cluster:
So zookeeper is based on Java development environment then need to install the Java installation package used here zookeeper version zookeeper-3.4.14, the installation package version Kafka was kafka_2.11-2.2.0.
AMQP protocol: Advanced Message Queuing Protocol (Advanced Message Queuing Protocol) is an open standard messaging middleware application layer protocol. AMQP defines the format of the byte stream data transmitted through a network. Therefore, compatibility is very good, any program can achieve AMQP protocol and other programs compatible with the AMQP protocol interaction, it can be done easily cross-language, cross-platform.
server1：192.168.42.128
server2：192.168.42.129
server3：192.168.42.130
First check before installing the system does not come with open-jdk
command:
rpm -qa | grep java
rpm -qa | grep jdk
If there is no input information is not installed.
Retrieve a list of 1.8
yum list java-1.8 *
All files are installed 1.8.0
yum install java-1.8.0-openjdk * -y
Check whether the installation was successful by using the command
java -version
cat /etc/hosts
192.168.42.128 kafka01
192.168.42.129 kafka02
192.168.42.130 kafka03
1. Turn off selinux, firewall.
setenforce 0
systemctl stop firewalld && systemctl disable firewalld
2. Create a directory of zookeeper and Kafka:
cd /usr/local/
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.0/kafka_2.11-2.2.0.tgz
tar xzvf zookeeper-3.4.14.tar.gz
xzvf tar kafka211220tgz
mkdir -pv zookeeper / {zkdata, zkdatalog} #zkdata snapshot log is located, zkdatalog transaction log is located
mkdir -pv kafka / kfdatalogs #kfdatalogs message log is located
4. Build and change the zookeeper profile (on the need to set up three servers):
# Zoo_sample can be understood as a zookeeper comes profile template, a copy of a file ending .cfg configuration.
cp -of /usr/local/zookeeper/conf/{zoo_sample,zoo.cfg}
tickTime = heartbeat time between 2000 #zookeeper server.
initLimit = 10 #zookeeper maximum connection time to failure
syncLimit = synchronous communication time of 5 #zookeeper
dataDir = / usr / local / zookeeper / absolute path to store a snapshot of the log zkdata #zookeeper
Absolute path to store transaction log of dataLogDir = / usr / local / zookeeper / zkdatalog #zookeeper of
clientPort = 2181 #zookeeper connection port and the client
server.1 = 192.168.11.139: 2888: 3888 # server and its number, server IP address, communication port, the port election
server.2 = 192.168.11.140: 2888: 3888 # server and its number, server IP address, communication port, the port election
server.3 = 192.168.11.141: 2888: 3888 # server and its number, server IP address, communication port, the port election
# The above ports are zookeeper default port, can be modified with demand
5. Create myid file:
On server1
echo “1” > /usr/local/zookeeper/zkdata/myid
# Is on a different server, the server number to send myid under zkdata.
6. Start zookeeper cluster
cd /usr/local/zookeeper/bin
./zkServer.sh start
./zkServer.sh status
#Mode: leader master node, Mode: follower from the node, usually only one cluster ZK Leader, a plurality of follower, the main response is generally read and write requests of the client, and synchronization data from the master, when the master will hang from the follower a leader in the vote out.
zookeeper electoral process / Works
https://blog.csdn.net/wyqwilliam/article/details/83537139
In zookeeper cluster, each node there are the following three kinds of roles and four states:
Role: leader, follower, observer
状态：leading，following，observing，looking
Each Server There are four states in the course of their work:
LOOKING: Current Server does not know who is the leader, we are searching for.
LEADING: Current Server is the elected leader.
FOLLOWING: leader has been elected, the current Server to synchronize.
OBSERVING: observer act exactly the same in most cases with the follower, but they do not participate in the elections and vote, but only to accept the results (observing) elections and voting
This, zookeeper cluster build end, based on the following zookeeper build kafka cluster:
Kafka's basic concepts:
Topic: Topic especially Kafka message source process (feeds of messages) different classification.
Partition: the packet on the physical Partition Topic, a topic can be divided into a plurality of partition, each partition is an ordered queue. partition each message is assigned a sequential id (offset).
Message: news, is the basic unit of communication, each producer can publish some messages to a topic (theme).
Producers: data producers message, called the producers to process a topic Kafka published message.
Consumers: Consumers of data messages, subscribe to topics and process messages issued by its process is called consumers.
Broker: caching proxy, Kafka cluster of one or more servers collectively referred to as broker, here is the AMQP protocol.
cd /usr/local/kafka/config/
1. Modify server.properties file:
broker.id = 1 # myid here and in the same file zookeeper, using the unique identification
prot = 9092 #Kafka between ports connected to the cluster, not in the profile, but the default is 9092, can be modified with the demand, here we add
log.dirs = / usr / local / kafka / kfdatalogs # Kafka message log stored absolute path
advertised.listeners=PLAINTEXT://kafka01:9092
log.retention.hours = 168 # default maximum message persistence time of 168 hours, 7 days
message.max.byte = 5242880 # saved messages maximum 5M
default.replication.factor = the number of copies saved messages 2 #kafka, if one copy fails, the other can continue to provide services
replica.fetch.max.bytes = 5242880 # take the maximum number of messages directly
zookeeper.connect = 192.168.42.128: 2181,192.168.42.129: 2181,192.168.42.130: IP address and zookeeper 2181 # various nodes of the cluster of ports, port zookeeper cluster setup is the number of port here is how much.
Unmodified profile information:
The number of threads num.network.threads = 3 # This is borker network processing
num.io.threads = 8 # borker This is the I / O processing threads
num.partitions = 1 # default number of partitions, a topic a default partition number
log.retention.hours = 168 # default maximum message persistence time of 168 hours, 7 days
message.max.byte = 5242880 # saved messages maximum 5M
default.replication.factor = the number of copies saved messages 2 #kafka, if one copy fails, the other can continue to provide services
replica.fetch.max.bytes = 5242880 # take the maximum number of messages directly
log.segment.bytes = 1073741824 # The parameters are: kafka because the messages are in the form of an additional floor to the file, when more than this value, kafka will play a new file
log.retention.check.interval.ms = 300000 # 300,000 every millisecond to check the configuration of the above log time to failure (log.retention.hours = 168), to the directory to see if there is expired messages if there are, delete
log.cleaner.enable = false # whether to enable log compression, generally do not enable, enabled, then you can improve performance
2. Start Kafka cluster:
cd /usr/local/kafka/bin
bash kafka-server-start.sh -daemon …/config/server.properties
3. Starting test:
3.1 Creating topic
./kafka-topics.sh --create --zookeeper 192.168.21.241：2181 --replication-factor 2 --partitions 1 --topic wg01
# -Replication-factor 2 two copies
# -Partitions 1 to create a partition
# -Topic tian tian theme
3.2 Creating a producter:
./kafka-console-producer.sh --broker-list 192.168.21.241:9092 --topic wg01
3.3 Creating a consumer:
./kafka-console-consumer.sh --bootstrap-server 192.168.11.141:9092 --topic wg01 --from-beginning
3.4 View topic:
--list --zookeeper 192.168.11.141:2181 ./kafka-topics.sh
----------------
Disclaimer: This article is original CSDN bloggers "A large abandoned ❀" of article, follow the CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/TH_lsq/article/details/102626967