Kafka is a distributed, partitionable, and replicable messaging system. It provides the functions of an ordinary messaging system, but has its own unique design. What does this unique design look like?
Introduction
Kafka is a distributed, partitionable, and replicable messaging system. It provides the functions of an ordinary messaging system, but has its own unique design. What does this unique design look like?
First let's look at a few basic messaging system terms:
• Kafka summarizes messages in topic units.
• The program that publishes messages to Kafka
topics becomes producers. • The program that subscribes to topics and consumes messages becomes consumers.
• Kafka runs as a cluster and can be composed of one or more services, each service is called a broker.
Producers pass The network sends messages to the Kafka cluster, and the cluster provides messages to consumers, as shown in the following figure:
The client and server communicate via TCP protocol. Kafka provides a Java client and supports multiple languages.
Description:
Operating system: CentOS 6.x 64-bit
Kafka edition: kafka_2.11-0.8.2.1
achieved goals:
Single machine installation and configuration kafka
Specific operation:
1. Turn off the firewall
Close: service iptables stop
disable: chkconfig iptables off
Second, install the JDK
Kadka operation requires JDK support
1. Download JDK
http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm
Note: Copy directly to the download tool for download. Please use JDK7 for the version. JDK8 may not be compatible with kafka_2.11-0.8.2.1
After the download is complete, upload it to the / usr / local / src directory
2. Install JDK
cd /usr/local/src
chmod + x jdk-7u79-linux-x64.rpm # add execute permission
rpm -ivh jdk-7u79-linux-x64.rpm #install
After the installation is complete, you can cd / usr / java / to the installation directory to view
3. Add JDK to system environment variables
vi / etc / profile #Edit, add the following code at the end
JAVA_HOME=/usr/java/jdk1.7.0_79
PATH=$PATH:$JAVA_HOME/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11R6/bin
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export JAVA_HOME
export PATH
export CLASSPATH
: wq! #Save and exit
source / etc / profile #make the configuration file effective immediately
java -version #View JDK version information
At this point, the JDK installation is complete.
Three, install kafka
1. Download kafka
cd /usr/local/src
wget http://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz
Note that the version of kafka_2.11-0.8.2.1.tgz is a compiled version and can be used after decompression.
tar -xzvf kafka_2.11-0.8.2.1.tgz #decompress
mv kafka_2.11-0.8.2.1 / usr / local / kafka #Move to the installation directory
2, arrangement kafka
mkdir / usr / local / kafka / log / kafka #Create kafka log directory
cd / usr / local / kafka / config #Enter the configuration directory
vi server.properties #Edit and modify the corresponding parameters
broker.id=0
port = 9092 #Port number
host.name = 192.168.5.56 #Server IP address, change to your own server IP
log.dirs = / usr / local / kafka / log / kafka #Log storage path, the directory created above
zookeeper.connect = localhost: 2181 #zookeeper address and port, stand-alone configuration deployment, localhost: 2181
: wq! #Save and exit
3. Configure zookeeper
mkdir / usr / local / kafka / zookeeper #create zookeeper directory
mkdir / usr / local / kafka / log / zookeeper #create zookeeper log directory
cd / usr / local / kafka / config #Enter the configuration directory
vi zookeeper.properties #Edit and modify the corresponding parameters
dataDir = / usr / local / kafka / zookeeper #zookeeper data directory
dataLogDir = / usr / local / kafka / log / zookeeper #zookeeper log directory
clientPort=2181
maxClientCnxns=100
tickTime = 2000
initLimit = 10
syncLimit=5
: wq! #Save and exit
Four, create a startup and shutdown kafka script
cd /usr/local/kafka
#Create startup script
vi kafkastart.sh #Edit, add the following code
#!/bin/sh
#Start zookeeper
/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3 #Wait after 3 seconds
#Start kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
: wq! #Save and exit
#Create close script
vi kafkastop.sh #Edit, add the following code
#!/bin/sh
#Close zookeeper
/usr/local/kafka/bin/zookeeper-server-stop.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3 #Wait after 3 seconds
#Close kafka
/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties &
: wq! #Save and exit
#Add script execution permission
chmod +x kafkastart.sh
chmod +x kafkastop.sh
Five, set the script to automatically execute at startup
vi /etc/rc.d/rc.local #Edit, add a line at the end
sh /usr/local/kafka/kafkastart.sh & # Set the script to automatically run in the background at boot
: wq! #Save and exit
sh /usr/local/kafka/kafkastart.sh #Start kafka
sh /usr/local/kafka/kafkastop.sh #Close kafka
At this point, Kafka stand-alone installation and configuration under Linux is complete.
Kafka is a distributed, partitionable, and replicable messaging system. It provides the functions of an ordinary messaging system, but has its own unique design. What does this unique design look like?
Introduction
Kafka is a distributed, partitionable, and replicable messaging system. It provides the functions of an ordinary messaging system, but has its own unique design. What does this unique design look like?
First let's look at a few basic messaging system terms:
• Kafka summarizes messages in topic units.
• The program that publishes messages to Kafka
topics becomes producers. • The program that subscribes to topics and consumes messages becomes consumers.
• Kafka runs as a cluster and can be composed of one or more services, each service is called a broker.
Producers pass The network sends messages to the Kafka cluster, and the cluster provides messages to consumers, as shown in the following figure:
The client and server communicate via TCP protocol. Kafka provides a Java client and supports multiple languages.
Description:
Operating system: CentOS 6.x 64-bit
Kafka edition: kafka_2.11-0.8.2.1
achieved goals:
Single machine installation and configuration kafka
Specific operation:
1. Turn off the firewall
Close: service iptables stop
disable: chkconfig iptables off
Second, install the JDK
Kadka operation requires JDK support
1. Download JDK
http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.rpm
Note: Copy directly to the download tool for download. Please use JDK7 for the version. JDK8 may not be compatible with kafka_2.11-0.8.2.1
After the download is complete, upload it to the / usr / local / src directory
2. Install JDK
cd /usr/local/src
chmod + x jdk-7u79-linux-x64.rpm # add execute permission
rpm -ivh jdk-7u79-linux-x64.rpm #install
After the installation is complete, you can cd / usr / java / to the installation directory to view
3. Add JDK to system environment variables
vi / etc / profile #Edit, add the following code at the end
JAVA_HOME=/usr/java/jdk1.7.0_79
PATH=$PATH:$JAVA_HOME/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/X11R6/bin
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export JAVA_HOME
export PATH
export CLASSPATH
: wq! #Save and exit
source / etc / profile #make the configuration file effective immediately
java -version #View JDK version information
At this point, the JDK installation is complete.
Three, install kafka
1. Download kafka
cd /usr/local/src
wget http://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.11-0.8.2.1.tgz
Note that the version of kafka_2.11-0.8.2.1.tgz is a compiled version and can be used after decompression.
tar -xzvf kafka_2.11-0.8.2.1.tgz #decompress
mv kafka_2.11-0.8.2.1 / usr / local / kafka #Move to the installation directory
2, arrangement kafka
mkdir / usr / local / kafka / log / kafka #Create kafka log directory
cd / usr / local / kafka / config #Enter the configuration directory
vi server.properties #Edit and modify the corresponding parameters
broker.id=0
port = 9092 #Port number
host.name = 192.168.5.56 #Server IP address, change to your own server IP
log.dirs = / usr / local / kafka / log / kafka #Log storage path, the directory created above
zookeeper.connect = localhost: 2181 #zookeeper address and port, stand-alone configuration deployment, localhost: 2181
: wq! #Save and exit
3. Configure zookeeper
mkdir / usr / local / kafka / zookeeper #create zookeeper directory
mkdir / usr / local / kafka / log / zookeeper #create zookeeper log directory
cd / usr / local / kafka / config #Enter the configuration directory
vi zookeeper.properties #Edit and modify the corresponding parameters
dataDir = / usr / local / kafka / zookeeper #zookeeper data directory
dataLogDir = / usr / local / kafka / log / zookeeper #zookeeper log directory
clientPort=2181
maxClientCnxns=100
tickTime = 2000
initLimit = 10
syncLimit=5
: wq! #Save and exit
Four, create a startup and shutdown kafka script
cd /usr/local/kafka
#Create startup script
vi kafkastart.sh #Edit, add the following code
#!/bin/sh
#Start zookeeper
/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3 #Wait after 3 seconds
#Start kafka
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &
: wq! #Save and exit
#Create close script
vi kafkastop.sh #Edit, add the following code
#!/bin/sh
#Close zookeeper
/usr/local/kafka/bin/zookeeper-server-stop.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3 #Wait after 3 seconds
#Close kafka
/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties &
: wq! #Save and exit
#Add script execution permission
chmod +x kafkastart.sh
chmod +x kafkastop.sh
Five, set the script to automatically execute at startup
vi /etc/rc.d/rc.local #Edit, add a line at the end
sh /usr/local/kafka/kafkastart.sh & # Set the script to automatically run in the background at boot
: wq! #Save and exit
sh /usr/local/kafka/kafkastart.sh #Start kafka
sh /usr/local/kafka/kafkastop.sh #Close kafka
At this point, Kafka stand-alone installation and configuration under Linux is complete.