First, the preparatory work
step1: install JDK1.8
step2: stand-alone installation zookeeper
step3: Installation Gradle-4.10.2
step4: Installation scala-2.11.12
Second, the source code is deployed to kafka editor and test IDEA
step1: will kafka Source deployed to IDEA
step2: source code compiler implementation of Kafka
Third, the modified source code release of Kafka into binary version
step1: Run
First, the preparatory work
step1: installation JDK1.8
installed on jdk is relatively simple, but it needs attention configure the environment variables
step2: zookeeper stand-alone installation
because kafka needs to be run on the basis of the zookeeper, so we only need a simple installation stand-alone version can zookeeper.
Directory structure is as follows:
- E: \ ZooKeeper \ Data
- E: \ ZooKeeper \ ZooKeeper-3.4.12
1. Download zookeeper 3.4.12 Download
2. decompress, depending on the storage location is E: \ zookeeper \ zookeeper- 3.4.12
3. Go to the directory E: \ zookeeper \ zookeeper-3.4.12 \ conf modify the configuration file, copy the file and rename zoo_sample.cfg zoo.cfg, add the following code.
E = dataDir: \\ \\ ZooKeeper Data
server.1 = your the IP: 2888: 3888
12 is
4. Double-bin folder in zkServer.cmd start ZooKeeper
5. The bin will be added to the system directory folder path in
step3: mounting Gradle -4.10.2
1 download gradle Download, attention to the selection-only binary
2 unzip and add bin folder directory to the system path in
step4: 2.11.12-install scala
1. Download scala Download, attention can be downloaded .msi install windows file format, directly after downloading the prompts to install (like a normal app install approach)
2. Add the bin folder in the directory to the system path
Second, the source code is deployed to kafka IDEA editor and test
step1: IDEA to deploy kafka source
editor is used for this IntelliJ IDEA editor, on the particular method of cloning used the source code directly from git.
Specific cloning Reference source blog
then performed in the Terminal IDEA Editor: gradle idea
of course be their offline download source, then the source into the root directory, Run gradle idea may, be introduced into the last file that is compiled IDEA can.
After introduction IDEA directory structure as follows:
The main part is the content source core module
Module Name
Description
ADMIN
Kafka used to live in the administrator module, operate and manage their topic, partition-related, include creating, deleting topic, or expand partitions.
api
is responsible for data exchange, encoding and decoding client and server interaction data.
cluster
here comprising a plurality of entity class, there Broker, Cluster, Partition, Replica. Wherein a plurality Broker Cluster composition, comprising a plurality of Partition Broker, all Topic Partition a different distribution of the Broker, a Replica contains both a Partition.
common
This is a common module, which comprises only the various errors and exception-based authentication.
consumer
consumer processing module, the client is responsible for all consumer data and logic processing.
controller
elections this module is responsible for the central controller, Leader election district, Replica of allocation or reallocation, partitions, and a copy of the expansion and so on.
coordinator
responsible for managing the part of the consumer group and their offset.
log
This is a file storage module responsible for Kafka, it is responsible for reading and writing data message Topic all of Kafka.
message
encapsulating a plurality of data composed of a set of data or the compressed data set.
metrics
responsible for monitoring the internal state of the module.
network
This module is responsible for handling and receiving client is connected, the processing time of the network module.
security
responsible for Kafka secure authentication and management module.
serializer
to serialize and deserialize the current message content.
server
contents of the module more involved with Leader and Offset the checkpoint, dynamic configuration, creation and deletion delay Topic, Leader elections, Admin and Replica management, as well as a variety of cache metadata and other content.
tools
read the module is a tool modules address will be more. There are offset value corresponding to the export of consumer; Export LogSegments information, as well as information on the current Topic Location of log write; export content offset value, etc. on the Zookeeper.
utils
variety of tools, such as Json, ZkUtils, thread pool tools, KafkaScheduler public scheduler class, Mx4jLoader monitor loader, ReplicationUtils replication set tools, CommandLineUtils contents of the command-line tools, as well as public logging and so on.
step2: compiler implementation of Kafka source
1. First Double-click the E: \ zookeeper \ zkServer.cmd under the zookeeper-3.4.12 \ bin directory to start zookeeper (execution box can not be closed, otherwise shut down the service), the results are as follows.
2. Install the plug-in idea of scala, Settings-> plugins-> scala view version 2018.2.11 ensure gradle.properties profile scalaVersion consistent installation
3. Use log4j output log kafka server, before the start of needs the placement log4j.properties configuration file to the core \ src \ main \ under the scala path, then if the compiler still can not print log information, a copy of the file to the core \ out \ production \ classes \ directory, and then run the program, so in order to correct the output log information can be obtained from this log4j.properties file config directory.
4. Modify kafka / config / server.properties file, add or modify the following two codes.
# Choose a suitable output log folder can pay attention to the new folder in the window system is best not to name spaces, such as "IDEA workspace", the extra space may cause IDEA happen java.nio.file. NoSuchFileException: F: IDEA, in fact, is named after the space among the contents of error will not recognize the file can not be found appears to be a pit, remember
log.dirs = F: \\ \\ log IDEA
# Please will you present yourself IP address of the machine to fill
zookeeper.connect = your IP: 2181
1234
5 configuration run / debug configuration
click run-> run / debug configrations or edit configrations
Click +, select Application
following configuration:
6. Click the IDEA run, performed separately kafka, consumer, producer, while the input data producer console , you can receive the consumer's console.
Third, the modified Kafka into binary version of the source distribution
in general, we cluster linux operating Kafka, using a binary version, respectively, and divided into Scala2.11 Scala2.12 version, as shown in FIG.
step1: execute commands
executed IDEA editor in Terminal:
gradle wrapper
gradlew releaseTarGz或者./gradlew releaseTarGz
gradle wrapper命令：
下载 wrapper 包，命令执行成功后，会在 kafka 源码包的 gradle 目录下生 成一个 wrapper 目录，其中包括gradle-wrapper.jar和gradle-wrapper.properties两个文件
gradlew releaseTarGz命令：
编译完之后将会在core/build/distributions/里面生成 kafka_2.12-2.2.0-SNAPSHOT.tgz 压缩包文件，这个和从网上下载的一样，可以直接用。
四、错误总结
错误一
出现：SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
解决：参考错误提示网址
具体做法：
1.下载相应的jar包，slf4j-log4j12.jar + Log4j.jar
2.导入jar包
具体的IDEA操作如下：
（1）File -> Project Structure -> Modules
（2）找到 core，core_main，打开 dependencies，点击 +，添加 sl4j-log4j12.jar、log4j. jar
错误二
出现：在producer的控制台上输入但是在consumer上没有收到消息
解决：端口和防火墙的问题
具体做法：
1.保证计算机的防火墙关闭
2.在设置run/debug configurations的时候需要保证端口相同，这个可以自己试试
错误三
出现：java.nio.file.NoSuchFileException: F:IDEA
解决：windows项目文件夹命名中带有空格
具体过程：原先的windows项目命名是IDEA worksapce，然后执行kafka的时候发现出现错误，并且提示找不到文件，因此将windows项目重命名为IDEA，再次执行，启动成功。
建议：文件夹的命名尽量使用英文并且不要带空格等不易识别的字符
错误四
出现：执行源码（即第二模块下的step2的第五步）出现错误，消费者无法执行
解决：这个问题与错误二类似，是填写的run/debug configures有问题
具体过程：请注意新版本的kafka和旧版本的kafka使用的命令是不一样的，例如旧版本的消费者使用--zookeeper参数指定zookeeper的地址，但是对于新版本的消费者需要使用--bootstrap-server参数指定broker的主机名和端口。
建议：关注kafka版本变化，使用对应版本的命令和功能。
错误五
出现：在ubuntu16.04上执行kafka_2.12-2.2.0-SNAPSHOT，出现如下错误
-bash: ./kafka-server-start.sh: /bin/bash^M: 解释器错误: 没有那个文件或目录
解决：参考博客，可以使用该博客的方法解决，但是以后会出现很多这样的情况，最后决定将所有的项目重新迁移到ubuntu系统上进行
具体过程：这个文件在Windows 下编辑过，在Windows下每一行结尾是\n\r，而Linux下则是\n
错误六
出现：将项目迁移到ubuntu系统上在idea编辑器上进行编译，执行命令gradle idea的时候出现如下错误
lala@lala-Lenovo:~/test/kafka$ gradle idea
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
FAILURE: Build failed with an exception.
* Where:
Build file '/home/lala/test/kafka/build.gradle' line: 57
* What went wrong:
A problem occurred evaluating root project 'kafka'.
> Failed to apply plugin [id 'org.owasp.dependencycheck']
> Could not create task of type 'Analyze'.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
出现这个的原因是在ubuntu上安装的gradle版本太低（使用命令sudo apt-get install gradle，安装的是2.7版本），但是kafka要求（参照）5.0以上，因此重新安装了gradle，执行成功
错误七
出现：执行run-kafka时候（即第二模块下的step2的第6步）,出现错误如下
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /logs/server.log (没有那个文件或目录)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
.......(省略)
at kafka.utils.Logging.$init$(Logging.scala:47)
at kafka.Kafka$.<init>(Kafka.scala:30)
at kafka.Kafka$.<clinit>(Kafka.scala)
at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [stateChangeAppender].
12345678910
解决：参考stack overflow
将log4j.propertties中有关路径的地方改成绝对路径即可
log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/logs/server.log //更改此行
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.stateChangeAppender.File=${kafka.logs.dir}/logs/state-change.log //更改此行
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.requestAppender.File=${kafka.logs.dir}/logs/kafka-request.log //更改此行
log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.cleanerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.cleanerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.cleanerAppender.File=${kafka.logs.dir}/logs/log-cleaner.log //更改此行
log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.controllerAppender.File=${kafka.logs.dir}/logs/controller.log //更改此行
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.authorizerAppender.File=${kafka.logs.dir}/logs/kafka-authorizer.log //更改此行
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
1234567891011121314151617181920212223242526272829303132333435
错误八
出现：执行gradlew releaseTarGz时候,出现错误如下
gradlew releaseTarGz
未找到 'gradlew' 命令，您要输入的是否是：
命令 'gradle' 来自于包 'gradle' (universe)
gradlew：未找到命令
12345
Solution: The command into ./gradlew releaseTarGz
reasons: different versions of command under the windows command and Linux
---------------------
Author: MAS- QUE-RADE
source: CSDN
original: https: //blog.csdn.net/u012606565/article/details/83022630
copyright: This article is a blogger original article, reproduced, please attach Bowen link!