First, log collection system background
1 Background
- a. Each system has a log when system problems, need to be addressed through the log
- b. When the system is relatively little time machine, log in to view the server to meet
- c. When a machine the size of a huge system, log in to the machine almost unrealistic view
2 Solutions
- a. Log on to the machine to collect real-time, unified storage system to the center
- b. and then indexed to these logs, which can be found by searching for the corresponding log
- c. By providing a user-friendly web interface, thus completing the log through the web search
3 problems encountered
a. real-time log is very large, billions of messages per day
b. log near real-time collection, delay in minutes to control the level of
c. horizontally scalable
4 ELK industry program
5 ELK Programs
a. high operation and maintenance costs for each additional log collection, you need to manually modify the configuration
b. control deficiencies, the state can not obtain accurate logstash
c. can not do custom development and maintenance
Second, the log collection system architecture
6 log collection system design
Introduction of the components
a. Log Agent, log collection client to collect the log on the server
b. Kafka, high throughput distributed queue, linkin development, apache top open source projects
c. ES, elasticsearch, open-source search engine that provides http-based the web interface restful
d. Hadoop, distributed computing frameworks, large amounts of data can be distributed processing platform