Kafka uses Zookeeper to save cluster metadata information and consumer information .
The installation directory is /usr/local/zookeeper-3.4.11
Unzip:
[root@U10-33 local]# tar -zxf zookeeper-3.4.11.tar.gz
Configuration file ( zoo.cfg ):
[root@U10-33 ~]# cd /usr/local/zookeeper-3.4.11/conf/ [root@U10-33 conf]# ll Total dosage 16 -rw-r--r-- 1 502 games 535 11月 2 02:47 configuration.xsl -rw-r--r-- 1 502 games 2161 November 2 02:47 log4j.properties -rw-r--r-- 1 root root 1005 3月 23 17:21 zoo.cfg -rw-r--r-- 1 502 games 57 3月 23 14:42 zoo_sample.cfg [root@U10-33 conf]# vi zoo.cfg
tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181 # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit = 10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/var/lib/zookeeper/ #This directory is pre-created # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature # autopurge.purgeInterval = 1
Create the data directory ( /var/lib/zookeeper):
[root@U10-33 ~]# cd /var/lib [root@U10-33 lib]# mkdir zookeeper
Configure environment variables
[root@U10-33 ~]# vi /etc/profile
Add the following information:
ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.11 CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$ZOOKEEPER_HOME/lib: export PATH=$ZOOKEEPER_HOME/bin:$PATH
Load the variable just set
[root@U10-33 ~]# source /etc/profile
start up:
[root@U10-33 ~]# cd /usr/local/zookeeper-3.4.11/bin/ [root@U10-33 bin]# ll Total dosage 36 -rwxr-xr-x 1 502 games 232 11月 2 02:47 README.txt -rwxr-xr-x 1 502 games 1937 November 2 02:47 zkCleanup.sh -rwxr-xr-x 1 502 games 1056 11月 2 02:47 zkCli.cmd -rwxr-xr-x 1 502 games 1534 November 2 02:47 zkCli.sh -rwxr-xr-x 1 502 games 1628 11月 2 02:47 zkEnv.cmd -rwxr-xr-x 1 502 games 2696 November 2 02:47 zkEnv.sh -rwxr-xr-x 1 502 games 1089 11月 2 02:47 zkServer.cmd -rwxr-xr-x 1 502 games 6773 November 2 02:47 zkServer.sh [root@U10-33 bin]# ./zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.11/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
verify:
[root@U10-33 ~]# telnet localhost 2181 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. srvr Zookeeper version: 3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x0 Mode: standalone Node count: 4 Connection closed by foreign host.
You can now connect to the Zookepr port and verify that Zookeeper is installed correctly by sending the four-word command srvr.
[root@U10-33 ~]# telnet localhost 2181 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. srvr Zookeeper version: 3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x0 Mode: standalone Node count: 4 Connection closed by foreign host.--------------------------------------------------------------------------------------
Zookeeper group (Ensemble)
Zoo keeper clusters are called groups. Zookeeper uses a consensus protocol , so it is recommended that each group should contain an odd number of nodes (such as 3, 5, etc.), because only when the majority of nodes in the group (that is, a certain number of people) are available. , Zookeeper can process external requests . That is, if you have a group of 3 nodes, then it allows one node to fail. If the group contains 5 nodes, then it allows 2 nodes to fail.
Suppose there is a group of 5 nodes. If you want to make some configuration changes to the group, including replacing nodes, you need to restart each node in turn. If your swarm cannot tolerate multiple node failures, there is a risk when doing swarm maintenance. However, it is not recommended that a group contain more than 7 nodes , because Zoo keeper uses a consensus protocol, and too many nodes will reduce the performance of the entire group.
The group needs to have some common configuration, which lists all the servers, and each server also creates a myid file in the data directory to indicate its own ID. If the machine names of the servers in the group are zool, example.com, zoo2.example.com, zoo3.example.com, then the configuration file might look like this:
tickTime=2000
dataDir= /var/lib/zookeeper
clientPort=2181
initLimit =20
syncLimit=5
server.1=zool.example.com:2888:3888
server.2=zoo2.example.com:2888:3888
server.3=zoo3.example. com:2888:3888
initLimit represents the time limit for establishing an initial connection between the slave node and the master node,
syncLimit indicates the upper limit of the time allowed for the slave node to be out of sync with the master node,
Both values are multiples of tickTime, so initLimit=20*2000ms, which is 40s.
The server address follows the format of server.X=hostname:peerPort:leaderPort. Parameter description :
x The D of the server must be an integer, and it does not have to start from 0, nor does it need to be continuous;
the machine name or IP address of the hostnam server:
peerPort TCP port used for inter-node communication;
leaderPort The TCP port used for leader election.
The client only needs to connect to the group through the clientPort,
and the communication between the group nodes needs to use the three ports (peerPort, leaderPort, clientPort) at the same time.
----------------------------------------------------------------------------
kafka dress:
Unzip (/usr/local/kafka_2.12-1.0.1):
[root@U10-33 local]# tar -zxf zookeeper-3.4.11.tar.gz
3) Configuration ( server.properties ):
[root@U10-33 ~]# mkdir /tmp/kafka-logs #Save message log
[root@U10-33 config]# pwd /usr/local/kafka_2.12-1.0.1/config [root@U10-33 config]# vi server.properties
Add to
listeners=PLAINTEXT://localhost:9092 log.dirs=/tmp/kafka-logs broker.id=1
start kafka
[root@U10-33 config]# /usr/local/kafka_2.12-1.0.1/bin/kafka-server-start.sh -daemon /usr/local/kafka_2.12-1.0.1/config/server.properties
Once Kafka is created, you can do some simple things with the cluster to verify that it is installed correctly, such as creating a test topic, publishing some messages, and then reading them.
Create and verify the subject:
[root@U10-33 ~]# /usr/local/kafka_2.12-1.0.1/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Created topic "test".
View all topics:
[root@U10-33 ~]# /usr/local/kafka_2.12-1.0.1/bin/kafka-topics.sh --list --zookeeper localhost:2181 test
View the specified topic
[root@U10-33 ~]# /usr/local/kafka_2.12-1.0.1/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Start the KAFKA producer and publish a message on the test topic:
[root@U10-33 ~]# /usr/local/kafka_2.12-1.0.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --sync --topic test
Read messages from the test topic:
[root@U10-33 ~]# /usr/local/kafka_2.12-1.0.1/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[2018-03-23 18:46:32,381] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$)
java.net.UnknownHostException: U10-33: U10-33: unknown error
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:135)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:159)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:112)
at kafka.consumer.OldConsumer.<init>(BaseConsumer.scala:130)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:72)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: java.net.UnknownHostException: U10-33: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 7 more