introduction
In traditional project, if in a production environment, a number of different clusters of servers, if required by the production environment Bug log positioning of the project, the need to query using the traditional way of command on each node, so the efficiency is very underneath.
Typically, the log is dispersed on different storage devices. If you manage tens of hundreds of servers, you log on each machine in turn still using the traditional method of inspection of the log. This is not feeling very cumbersome and inefficient. Imperative that we use centralized log management, such as: open source syslog, log on all servers collect aggregate.
After the centralized log management, log statistics and retrieval has become a troublesome thing, we generally use grep, awk, and wc and other Linux commands can achieve retrieval and statistics, but for queries, sorting and statistical requirements more demanding and a large number of machines still use this method was somewhat powerless.
Deal with the problem of distributed log collection, the most popular way to use the ELK, in terms of the following took off.
1.ELK distributed log collection system
What is ELK? In fact ElasticSearch, Logstash, Kibana abbreviations, as follows:
-
" E lasticSearch" is based on a distributed Lucene open source search server. Its features include: distributed, zero-configuration, auto-discovery, auto-slice index, index replication mechanism, restful style interfaces, multiple data sources, automatically search load so on. It provides a distributed multi-user capabilities of full-text search engine, based on RESTful web interfaces . Elasticsearch is written in Java, and as open source under the Apache license terms published, is the second most popular search engine company. Designed for the cloud, to achieve real-time search, stable, reliable, fast and easy to install .
-
" L ogstash" is a fully open source tool that can log collect, filter, analyze, support a large number of data acquisition methods , and stored for later use for (eg search) . When it comes to search, logstash with a web interface, search and display all the logs. General work of c / s architecture, client installed on the host side need to collect logs, server side is responsible for each node the received log is filtered, modification and other operations in a concurrent to elasticsearch up.
-
" K ibana" is based Elasticsearch front-end presentation tool browser page, is an open source and free tools, log analysis Kibana may provide for Logstash and ElasticSearch friendly Web interface that can help you summarize, analyze and search for important data log .
2.Logstash Introduction
Logstash is a fully open source tool that can log collect, filter, analyze, support a large number of data acquisition methods, and store it for later use (such as search).
Logstash with a web interface, search and display all the logs. General work of c / s architecture, client installed on the host side need to collect logs, server side is responsible for each node the received log is filtered, modification and other operations in a concurrent to elasticsearch up.
Logstash event handler has three core processes: inputs → filters → outputs(receiving, processing, forwarding log). Support system log, webserver logs, error logs, application logs, in short, can throw to include all types of logs.
3.ELK distributed log collection system principle
Log collection principle:
- Each server node cluster installation Logstash log collection system plug.
- Each server node will enter into Logstash log in.
- Logstash the log format json formatted according to different indexes created daily output to the ElasticSearch.
- Kibana browser uses the query log is installed.
Installation Environment:
- Installation ElasticSearch (Reference: "Installing Linux environment Elasticsearch" )
- Installation Logstash (hereinafter speaking)
- Installation Kibana (Reference: "Linux environment to install Kibana" )
