再探kafka+flume实战

命令

启动zookeeper

启动两台机器上的kafaka

nohup kafka-server-start.sh  $KAFKA_HOME/config/server.properties &

这里的

启动kafka的消费端

kafka-console-consumer.sh --zookeeper hadoop01:2181 --topic test

启动flume端(kafka是两个机器都要启动，flume只要启动一个机器上的就行。注意kafka和flume的区别）

 flume-ng agent -n a1 -c conf -f $FLUME_HOME/conf/exec.conf -Dflume.root.logger=INFO,console

其中exec.conf配置如下

a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /export/data/flume_sources/click_log/1.log
a1.sources.r1.channels = c1

a1.channels.c1.type=memory
a1.channels.c1.capacity=10000
a1.channels.c1.transactionCapacity=100

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink #这里

org.apache.flume.sink.kafka.KafkaSink 对应的是flume里的一个jar包

a1.sinks.k1.topic = testa1.sinks.k1.brokerList = hadoop01:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel = c1

然后  echo "fdfsafjasdlfjlas" >>/export/data/flume_sources/click_log/1.log  

观察 消费端的变化