1.前提条件
CDH环境已经搭建成功。
2.搭建要求
在CDH上集成Kafka，要求用CDH上的Zookeeper管理Kafka而不是Kafka自带的Zookeeper。
3.搭建流程
(1)首先选取适合自己CDH版本的Kafka的Parcel文件进行下载，CDH与Kafka版本对应关系链接为：
https://www.cloudera.com/documentation/kafka/1-4-x/topics/kafka_packaging.html
(2)下载Kafka Parcel，下载链接为：http://archive.cloudera.com/kafka/parcels/1.4.0/
查看Linux服务器版本
#cat /proc/version
Linux version 3.10.0-693.2.2.el7.x86_64 ([email protected])
(gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Sep 12 22:26:13 UTC 2017
(3)上传Parcel文件至cloudrea目录，例如我的目录为：/DATA/cloudera/parcel-repo
(4)分配并激活Parcel，在cloudrea manager中选择主机–>Parcel，可以看到(3)中上传的Parcel包。
点击分配，分配完成后点击激活。(我的已经激活过故显示已分配已激活)
(5)手动修改配置文件(也可通过CM界面进行配置)
激活完成后，进入Kafka配置文件目录：/DATA/cloudera/parcels/KAFKA-0.8.2.0-1.kafka1.4.0.p0.56/etc/kafka/conf.dist
编辑server.properties文件，修改内容如下：
broker.id=1，其他两个为2、3
port=9092，其他两个为9093、9094
host.name=xx.xx.xx.xx，每个配置文件的这项改为本服务器的实际IP/hostname
zookeeper.connect=xx.xx.xx.xx:2181,xx.xx.xx.xx:2181,xx.xx.xx.xx:2181，此处填写实际zookeeper的配置信息。
(6)所有配置完成后，在CM中添加Kafka服务，添加服务时需要配置的两项如下：
Destination Broker List Kafka MirrorMaker Default Group
10.10.0.126:9092,10.10.0.127:9092,10.10.0.8:9092,10.10.0.27:9092
Source Broker List Kafka MirrorMaker Default Group
10.10.0.126:9092,10.10.0.127:9092,10.10.0.8:9092,10.10.0.27:9092
就是kafka所有节点的ip
kafka安装位置
/opt/cloudera/parcels/KAFKA-2.1.1-1.2.1.1.p0.18/bin
//创建topic
./kafka-topics --describe --zookeeper 10.10.0.27:2181 --topic bbb
//打开生产者客户端
./kafka-console-producer --broker-list 10.10.0.27:9092 --topic bbb
//打开消费者客户端
./kafka-console-consumer --zookeeper 10.10.0.27:2181 --topic bbb --from-beginning
//删除topic
./kafka-topics --delete --zookeeper 10.10.0.27:2181 --topic bbb
datanode2:9092,sparkmaster:9092,datanode1:9092
一定要看运维开的端口对不对！！！！
遇到的问题：
1.jar包冲突
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/E:/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/ms-notify/WEB-INF/lib/log4j-slf4j-impl-2.9.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/E:/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/ms-notify/WEB-INF/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
从日志里看到两个日志jar包冲突了：log4j-slf4j-impl-2.9.1.jar和slf4j-log4j12-1.7.2.jar
2.broker id不匹配
Opening socket connection to server sparkmaster/10.10.0.27:2181. Will not attempt to authenticate using SASL (unknown error)
2019-04-17 11:06:40,841 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.10.0.27:54562, server: sparkmaster/10.10.0.27:2181
2019-04-17 11:06:40,847 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server sparkmaster/10.10.0.27:2181, sessionid = 0x16a1fcddfc00ef0, negotiated timeout = 6000
2019-04-17 11:06:40,857 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
2019-04-17 11:06:40,895 INFO kafka.log.LogManager: Loading logs.
2019-04-17 11:06:40,901 INFO kafka.log.LogManager: Logs loading complete.
2019-04-17 11:06:40,986 INFO kafka.log.LogManager: Starting log cleanup with a period of 300000 ms.
2019-04-17 11:06:40,989 INFO kafka.log.LogManager: Starting log flusher with a default period of 9223372036854775807 ms.
2019-04-17 11:06:40,991 INFO kafka.log.LogCleaner: Starting the log cleaner
2019-04-17 11:06:40,993 INFO kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Starting
2019-04-17 11:06:40,997 FATAL kafka.server.KafkaServer: Fatal error during KafkaServer startup. Prepare to shutdown
找到配置文件:cat server.properties
修改broker.id
查看log.dirs所在位置
log.dirs=/home/user/tools/kafka/data
按着这个路径找到:meta.properties
修改brokerid