To develop a log monitoring system, it is necessary to comprehensively consider real-time, performance and stability
In order to cope with our volume and meet real-time performance, various open source technologies can be used for reference in the design and development of monitoring systems, flume, kafka, dubbo-esb (call chain identification), big data processing (spark, Hadoop)
Business logs also need to be monitored, and specifications need to be formulated
Design and Implementation of Distributed Log Stream Processing System Based on FlumeKafkaSpark
http://www.doc88.com/p-9955247066607.html
Kafka in action - Flume to Kafka
http://www.cnblogs.com/smartloli/p/4615908.html
Design and Implementation of Log Statistical Analysis System Based on Hadoop
http://www.docin.com/p-818403125.html
Study notes of Flume-ng+Kafka+storm
http://blog.csdn.net/zxcvg/article/details/18600335/
Flume-based Meituan Log Collection System (1) Architecture and Design
http://www.aboutyun.com/thread-8317-1-1.html
The best people to be recruited are:
1 Has a deep understanding of monitoring, has deeply participated in the development, design and implementation of large-scale monitoring systems, and is the core developer or architect
2. Familiar with various monitoring systems, know their advantages and disadvantages, and have ideas for integrating these systems
For QB: The monitoring system is constantly evolving and iterating to meet the needs. It is best to have a good foundation at the beginning and be able to use a set of ready-made things.