Kafka is installed and running on a Windows environment
Install
1. Install JDK
Regarding the installation of JDK and the configuration of environment variables, I will not go into details here. If you are unclear, you can view the JDK installation and environment variable configuration . (It is recommended to download Java officially released by Oracle, and the download address is: http://www.java.com/download/ ).
2. Install Zookeeper
First download the Zookeeper installation package from the Zookeeper official website .
After the download is complete, create a new bigData directory (D:\bigData) in the D drive as a directory for installing zookeeper and kafka, and directly decompress the zookeeper installation package. Note: It is best not to have spaces in the path, such as D:\Program Files, try not to use them, there will be problems when running the script.
3. Configure Zookeeper
a. Enter the file directory where the relevant settings of zookeeper are located, such as: D:\bigData\zookeeper-3.4.10\conf in this article, and rename zoo_sample.cfg to zoo.cfg. Open zoo.cfg and modify the configuration as follows:
Note: This is just to modify the configuration of dataDir=/tmp/zookeeper.
b. Configure the zookeeper environment variables:
ZOOKEEPER_HOME=D:\bigData\zookeeper-3.4.10
Edit the path variable in the system variable and add %ZOOKEEPER_HOME%\bin
At this point, zookeeper has been configured. Open cmd, enter zkserver, and run zookeeper. The results are as follows:
4. Afuka Kafka
First download the Kafka installation package from the Kafka official website http://kafka.apache.org/downloads . (To download the Binary downloads type, do not download the source file, it is convenient to use)
Modify the Kafka configuration file:
a. Modify the server.properties file in the config directory and modify log.dirs=D:/bigData/kafka_2.11-1.1.0/kafka-logs .
Note: In the server.properties file, zookeeper.connect=localhost:2181 represents the server IP and port where the zookeeper connected to kafka is located, which can be changed as needed. This article is used on the same machine, so it is not
with modification.
b. Modify the log4j.properties file in the config directory and modify log4j.appender.kafkaAppender.File=D:/bigData/kafka_2.11-1.1.0/logs/server.log
Other places do not need to be modified for the time being. Kafka will run on port 9092 according to the default configuration and connect to zookeeper's default port 2181.
run Kafka
Note : Before starting the kafka server, you must ensure that the Zookeeper instance is already running, because the operation of kafka requires a distributed application coordination service such as zookeeper.
Enter the kafka installation directory D:\bigData\kafka_2.11-1.1.0, press shift+right mouse button, select "Open command window here", open the command line, and enter in the command line: .\bin\windows\kafka -server-start.bat .\config\server.properties Enter. The normal startup interface is shown in the figure:
Note : The kafka startup under Windows reports an error that the main class Files\Java\jdk1.8.0_121\lib\dt.jar;C:\Program cannot be found or loaded.
Solution:
Open the file kafka-run-class.bat under the path D:\bigData\kafka_2.11-1.1.0\bin\windows, search for -cp %CLASSPATH%, and modify it to -cp "%CLASSPATH%" 
verify kafka
Create a topic (Topic)
- Create a topic named "test2018" with replicationfactor=1 (since there is only one kafka server running). The number of replicationfactors can be modified according to the number of kafka servers in the cluster to improve system fault tolerance.
- Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory and enter the command: kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 -- partitions 1 --topic test2018 Enter.
At this point, the topic is created, and you can view the directory corresponding to the topic in the kafka installation directory.
Create a producer
- Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory and enter the command: kafka-console-producer.bat --broker-list localhost:9092 --topic test2018 Press Enter. ( Do not close this window )
Create a consumer
- Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory and enter the command: kafka-console-consumer.bat --zookeeper localhost:2181 --topic test2018 and press Enter.
Now the producer and consumer have been created. Enter information in the producer command line window and observe the consumer command line window.
The above is the installation and basic use of kafka under windows.
In fact, the new version of Kafka already comes with zookeeper. Kafka uses zookeeper as its distributed coordination framework, which combines the processes of message production, message storage, and message consumption well. At the same time, with the help of zookeeper, kafka can establish a subscription relationship between producers and consumers, and achieve load balancing between producers and consumers in a stateless state for all components including producers, consumers and brokers.
Therefore, in the above steps, we can directly modify the zookeeper.properties file in Kafka's config directory ( D:\bigData\kafka_2.11-1.1.0\config) without installing zookeeper
#dataDir=/tmp/zookeeper
dataDir=D:/bigData/kafka_2.11-1.1.0/data/zookeeper
Open a new command line in the D:\bigData\kafka_2.11-1.1.0\bin\windows directory and enter the command: zookeeper-server-start.bat ../../config/zookeeper.properties and press Enter. You can also start zookeeper.