Collect the data of the file call.log to kafka, and obtain the data from the kafka consumer console.
Flume + kafka is a classic log collection tool for big data. File data is collected by flume, subscribed and published through kafka and cached, which is very suitable as a message middleware.
Ready to work
Start zookeeper, kafka cluster
./bin/zkServer.sh start
./bin/kafka-server-start.sh /config/server.properties
Create a ct theme in kafka, and set the number of partitions and the number of copies. These information will be saved on zookeeper.
./bin/kafka-topics.sh --zookeeper master:2181 --create --topic ct --partitions 3 --replication-factor 2
Start the Kafka console consumer, you can see the collected data in this process.
./bin/kafka-console-consumer.sh --zookeeper master:2181 --topic ct --from-beginning
Start flume, where the flume-kafka.conf configuration file is below.
./bin/flume-ng agent --conf ./conf/ --name a1 --conf-file ./flume-kafka.conf
flume-kafka.conf
# define
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F -c +0 /usr/local/data/call.log
a1.sources.r1.shell = /bin/bash -c
# sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092
a1.sinks.k1.kafka.topic = ct
a1.sinks.k1.kafka.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
# channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# bind
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
summary
- It can be found that the operation of kafka must pass through zookeeper, which can well understand the role of zookeeper in the kafka cluster.
- After running, the data of the file call.log will be sent to kafka, no matter which node, create a consumer through kafka, get the topic topic will get the data.
- Flume sink has direct kafka source, the two can be easily combined