Introduction
Kafka is the Apache Software Foundation to develop an open source stream processing platform, the Scala and Java to write. Kafka is a high throughput of distributed publish-subscribe messaging system, it can handle all the actions of consumers streaming data in the site. This action (web browsing, search and other user action) is a key factor in many social functions in modern networks. These data usually due to the required throughput is achieved by the polymerization process log and the log. For like Hadoop as the log data and off-line analysis systems, but requires real-time processing limitations, this is a viable solution. Kafka's purpose is to Hadoop parallel loading mechanism to unify online and offline messaging, but also in order to pass the cluster to provide real-time information.
A, KAFKA
Architecture Figure:
- Producer : Manufacturer, is transmitted in one message. Producers are responsible for creating a message, by zookeeper find Broker , and then deliver it to Kafka in.
- Consumer : consumer, that is, the party receiving the message. By zookeeper find the corresponding broker for consumption , and thus the corresponding service logic.
- Broker : Service Agent node. For Kafka terms, Broker may simply be seen as an independent Kafka service node or Kafka service instance. In most cases it can be Broker seen as a Kafka server, provided that this server only deployed a Kafka instance. One or more Broker form a Kafka cluster. Generally speaking, we are more accustomed to using the first letter lowercase broker to represent the service agent node
Send a message flow diagram:
Kafka multiple copies ( Replica ) mechanism :
As shown above in FIG Kafka cluster there . 4 th Broker , a topic has three partitions, and a copy of the factor (i.e. copy number) is also 3 , so each partition there . 1 th leader copies and 2 th follower copies . Producers and consumers only with the leader carried a copy of the interaction, and the follower replica is only responsible for synchronization messages, often follower relative copy of the message in the leader a copy of the terms there will be some lag.
Two, Zookeeper
ZooKeeper is a distributed , open-source distributed application coordination service, is Google a Chubby's open source implementation, is an important component of Hadoop and Hbase. It is to provide a consistent service for distributed applications, provides features include: configuration maintenance, domain name service, distributed synchronization, group services.
principle:
Chart can be high:
Each figure Server represents the server to install a Zookeeper services. Composition ZooKeeper server services will maintain current server state in memory, and maintained between each server communicate with each other. Between the cluster to maintain data consistency by Zab protocol (Zookeeper Atomic Broadcast).
Third, the deployment kafka & zookeeper cluster
We chose the official chart Address: https://github.com/helm/charts/tree/master/incubator/kafka
1) write your own values.yaml file (note my storageClass is already done the storage of nfs)
imageTag: "5.2.2"
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 1
memory: 2Gi
kafkaHeapOptions: "-Xmx2G -Xms2G"
persistence:
enabled: true
storageClass: "managed-nfs-storage"
size: "40Gi"
zookeeper:
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 100m
memory: 536Mi
persistence:
enabled: true
storageClass: "managed-nfs-storage"
size: "10Gi"
2) Installation kafka
Add chart Warehouse:
$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator
deploy
$ helm install --name kafka -f values.yaml incubator/kafka
Finally, we can see:
Fourth, the test kafka high availability
1) Follow the prompts to create a test client
apiVersion: v1
kind: Pod
metadata:
name: testclient
namespace: sscp-test
spec:
containers:
- name: kafka
image: solsson/kafka:0.11.0.0
command:
- sh
- -c
- "exec tail -f /dev/null"
Once you have the testclient pod above running, you can list all kafka
topics with:
kubectl -n sscp-test exec testclient -- kafka-topics --zookeeper kafka-test-zookeeper:2181 --list
To create a new topic:
kubectl -n sscp-test exec testclient -- kafka-topics --zookeeper kafka-test-zookeeper:2181 --topic test1 --create --partitions 1 --replication-factor 1
To listen for messages on a topic:
kubectl -n sscp-test exec -ti testclient -- for x in {1..1000}; do echo $x; sleep 2; done | kafka-console-producer --broker-list kafka-test-headless:9092 --topic test1
To stop the listener session above press: Ctrl+C
To start an interactive message producer session:
kubectl -n sscp-test exec -ti testclient -- kafka-console-producer --broker-list kafka-test-headless:9092 --topic test1
To create a message in the above session, simply type the message and press "enter"
To end the producer session try: Ctrl+C
Note: There are three kafka nodes, three copies of the message to be sent in order to maintain its high availability! ! !
Fifth, test Zookeeper High Availability
1.Create a node by command below:
“kubectl exec -it testclient bash -n sscp-test”
“zookeeper-shell kafka-test-zookeeper-headless:2181 create /foo bar”
2. Check zookeeper status
Watch existing members:
$ kubectl run --attach bbox --image=busybox --restart=Never -- sh -c 'while true; do for i in 0 1 2; do echo zk-${i} $(echo stats | nc kafka-zookeeper-${i}.kafka-zookeeper-headless:2181 | grep Mode); sleep 1; done; done'
zk-2 Mode: follower
zk-0 Mode: follower
zk-1 Mode: leader
zk-2 Mode: follower
3.kill the leader by command below:
“Kubectl delete pod kafka-test-zookeeper-1”
4.Check the previously inserted key by command below:
““kubectl exec -it testclient bash -n sscp-test”
“zookeeper-shell kafka-test-zookeeper-headless:2181 get /foo”
