Introduction
Apache Kafka is a free open source stream processing software platform, written by the Apache Software Foundation with Scala. It is a distributed news agency, dedicated to effectively deal with a large number of real-time information. Agent compared to other message systems (e.g. ActiveMQ and RabbitMQ), Apache Kafka higher throughput. Apache Kafka published to any number of systems based on real-time applications or submit a log that allows users to subscribe to the data.
Apache Kafka can be deployed on a single Web server can be deployed in a distributed cluster environment. Apache Kafka There are four main API: Producer API, Consumer API, Connector API and Streams API.
feature:
- Loading data into the parallel support;
- High throughput, even with modest hardware can support hundreds of thousands of messages per second;
- Persistent messaging disk structure and O (1), to provide a stable performance time, even several TB stored message;
- Distributed systems can be easily expanded without downtime.
This tutorial requires
- Install Ubuntu 16.04 cloud servers ECS platform;
- Configure a static IP address 192.168.0.103;
- Set Root password on the server.
Start cloud instance ECS
First, log cloud ECS console, you can choose your favorite cloud platform. ECS create a new instance, select Ubuntu 16.04 as the operating system has a 2GB RAM minimum. ECS is connected to the instance and login as root.
After logging in to Ubuntu 16.04 instance, run the following command to use the latest available package updates the basic system:
apt-get update -y
Install Java
Apache Kafka requires Java Runtime Environment, and therefore requires the latest version of Java on your system. By default, Ubuntu 16.04 repositories do not provide the latest version of Java. Therefore, we need to add the Java library into the system, you can do this by running the following command:
add-apt-repository ppa:webupd8team/java
Next, run the following command to update the repository and install the Java:
After installing Java, the Java version can be checked using the following command:
java -version
Output:
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
Install Zookeeper
Apache Kafka rely Zookeeper maintain configuration information, providing distributed synchronization, and providing group naming service. Therefore, we need to Zookeeper installed on your system, you can install it by running the following command:
apt-get install zookeeperd -y
By default, Zookeeper listens on port 2181, you can check it by running the following command:
netstat -nlpt | grep ':2181'
You should see the following output:
tcp6 0 0 :::2181 :::* LISTEN
Install Apache Kafka
First, you need can be downloaded from the Apache Web site to download the latest version of Kafka by running the following command:
wget http://redrockdigimark.com/apachemirror/kafka/1.1.0/kafka_2.12-1.1.0.tgz
Once downloaded, unzip the file using the following command:
xvzf tar kafka212110tgz
Next, copy the unzipped directory to / opt:
cp -r kafka_2.12-1.1.0 /opt/Kafka
Kafka script starts the server by running the following:
/opt/Kafka/bin/kafka-server-start.sh /opt/Kafka/config/server.properties
You should see the following output:
[2018-05-20 08:13:54,271] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2018-05-20 08:13:54,449] INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser)
[2018-05-20 08:13:54,461] INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser)
[2018-05-20 08:13:54,466] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
Kafka server is listening on port 9092.
Testing Apache Kafka
Now, by running the following command, using only one copy of a single partition and create the first theme named Topic1 of:
/opt/Kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Topic1
You should see the following output:
Created topic "Topic1".
Now, you can view the topic has been created on Kafka by running the following command:
/opt/Kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181
You should see the following output:
Topic1
Now, an example of a message using the following command to publish to Apache kafka theme called Topic1 of:
/opt/Kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Topic1
>>Hello Kafka
>How R You
>Ok
>
Next, run from Kafka Kafka Consumer cluster data read command and message to standard output:
/opt/Kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Topic1 --from-beginning
The following output should see the news release:
Hello Kafka
How R You
OK
Below about Kafka articles you might like, you may wish to refer to the following:
Details of Kafka : click here
Kafka download address : click here