java -> kafka -> logstash -> elasticsearch + kibana
①java writes logs to kafka
②logstash gets data from kafka
③ logstash writes data to elasticsearch
④kibana is used to display elasticsearch data
Use of the entire log architecture software version
kafka : kafka_2.10-0.10.0.0.tgz
logstash:logstash-2.3.0.tar.gz
elasticsearch: elasticsearch-2.3.0.tar.gz
kibana: kibana-4.3.0-linux-x64.tar.gz
Use log4j to write program logs to kafka in real time
http://www.cnblogs.com/gaopeng527/p/5266351.html
Kafka interacts with logstash data
http://blog.csdn.net/xiaoyu_bd/article/details/52454951
logstash
Logstash installation and usage
http://blog.csdn.net/u012373815/article/details/51029826
Logstash+elasticsearch configuration file creation rules
https://www.elastic.co/guide/en/logstash/current/configuration.html
Elasticsearch + kibana
The integration of kibana and Elasticsearch only requires a simple configuration file of Kibana
Kibana is an open source analytics and visualization platform designed for use with Elasticsearch
Installation and use of kibana4
http://blog.csdn.net/u012373815/article/details/51029855
The blog here is written in more detail: http://blog.csdn.net/ming_311/article/details/50619859
Kibana official example: https://my.oschina.net/clgo/blog/671861
Kafka producer consumption monitoring, input is sent to Elasticsearch and Kibana is used for front-end display.
Install the elasticsearch plugin elasticsearch-head
How to install elasticsearch-head plugin elasticsearch-2.2.0?
The Head plug-in is a graphical interface of Elasticsearch, which is convenient for users to add, delete, modify and query data, and interact with data in four ways based on REST.
1. Install and use kafka
Reference: http://colobu.com/2014/08/06/kafka-quickstart/
2. Install and use elasticsearch
Reference: http://shensuqiao.iteye.com/blog/2251674
3. Install the elasticsearch plugin elasticsearch-head
①https://github.com/mobz/elasticsearch-headDownload zip and decompress
②Create elasticsearch-2.3.1\plugins\head file
③ Copy all the files under the decompressed elasticsearch-head-master folder to the head (note: here are the files under the elasticsearch-head-master folder)
④Run es
⑤打开http://localhost:9200/_plugin/head/
Startup error:
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
解决：http://blog.csdn.net/u010317005/article/details/52205825
参考：
1.kafka入门教程
http://colobu.com/2014/08/06/kafka-quickstart/
创建只有一个备份一个分区的主题test
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
查看主题
bin/kafka-topics.sh --list --zookeeper localhost:2181
发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
消费消息
./kafka-console-consumer.sh --zookeeper 10.10.28.111:2181 --topic test --from-beginning
查看主题信息
bin/kafka-topics.sh --describe --zookeeper 10.10.28.111:2181 --topic test
kafka+log4j
http://www.cnblogs.com/dreammyle/p/6598935.html
http://stackoverflow.com/questions/35628706/kafka-log4j-appender-0-9-does-not-work
2.认识kibana
http://blog.csdn.net/ming_311/article/details/50619804
3.Elasticsearch教程
http://www.sojson.com/blog/85.html
4.跨域问题处理
5.网关编写