As the operation and maintenance engineers, we need to troubleshoot the server every day, you can help us locate the first problem is to view the server log, you can quickly locate the problem log. Currently we are talking about logging including system logs, application logs, and security logs. System operation and maintenance and server hardware and software developers can learn through the log information, check the cause of the error and the configuration process the error occurred. Often you need to analyze the log can understand the server load, performance, security, and thus take timely measures to correct the error. And log is stored dispersed on different devices.
If you manage several hundreds of servers, log on to the traditional methods we review the log of each machine. This is not feeling very cumbersome and inefficient. Imperative that we use centralized log management, such as: open source syslog, log on all servers collect aggregate.
After the centralized log management, log statistics and retrieval has become a troublesome thing, we generally use the find, grep, awk, and wc and other Linux commands can achieve retrieval and statistics, but for more demanding queries, sorting and statistics and other requirements and the huge number of machines still use this method was somewhat powerless. Today to share real-time log analysis ELK open-source platform we can perfect solution to the above problems, ELK by the ElasticSearch, Logstash and Kiabana three open source tools.
1) elasticsearch is an open source distributed search engine, its features are: distributed, zero-configuration, auto-discovery, indexing automatic fragmentation, index replication mechanism, restful style interfaces, multiple data sources, automatically search load, etc., ELK official website: HTTPS : //www.elastic.co/
2) Logstash is a fully open source tools, he can be your log collection, filtration, and store it for later use (such as search).
3) Kibana is also an open source and free tools that can analyze Kibana friendly Web interface and log Logstash ElasticSearch provided to help you summarize, analyze and search for important data logs.
4) Logstash and Elasticsearch is written in the Java language, and Kibana using node.js framework, in the configuration ELK environment to ensure that the system has JAVA JDK development library.
ELK principle topology
1) ELK workflow
logstash log data collection terminal APP customers, all the log filter out into Elasticsearch search engine, and then presented to the user through the WEB front-end Kibana GUI, users need to view the contents of the log can be specified by.
Communication may be simultaneously added redis queue:
Figure I;
Figure II;
2) After addition Redis queue workflow
Logstash include Index and Agent (shipper), Agent responsible for the client to monitor and filter the log, while the Index is responsible for collecting logs and logs to ElasticSearch, ElasticSearch local log storage, indexing, providing search, kibana can be obtained from ES cluster think to log information.