1 Background
In our daily life, we often need to review some of the things that happened before; or, if there are some problems, they can go to find out the reasons in some places, looking for traces of the problems. Inevitably need to use the text, and so on different forms of recording images. Expressed in term of a computer, it is the LOG, or log.
Logs, for any system, and are an important part. In a computer system inside, even more so. However, because most of the current computer system is more complex, many systems are not in one place, even all cross-border; even a system in place, there are a variety of sources, such as operating systems, applications services, business logic, etc. . They are constantly produces a wide variety of log data. According to incomplete statistics, about the world we want to generate data 2EB (1018) per day.
Faced with such vast amounts of data, and is distributed in various places, if we need to find some important information, are still using traditional methods, to log in to view on a single machine? It seems the traditional tools and methods have been a clumsy and inefficient. So, some smart people on the proposed establishment of a centralized method, the data from different sources into one centralized place.
A complete centralized logging system, is inseparable from the following main features.
- Collection - Log data can be collected from multiple sources
- Transmission - can stably transmit the log data to the central system
- Storage - How to store log data
- Analysis - analysis supports UI
- Warning - can provide error reporting, monitoring mechanisms
2 ELK Introduction
Elastic Stack is a collection of software Elasticsearch, Logstash, Kibana short, they are open source software. Added a FileBeat, it is a lightweight log collection processing tools (Agent), Filebeat small footprint, suitable for transmission on each server logs to collect after Logstash, the government has also recommended this tool.
- Elasticsearch based on java, it is a Lucene-based, full-text indexing support for distributed storage and indexing engine, is mainly responsible for the log index and stored for easy retrieval query the business side. Its features include: distributed, zero-configuration, auto-discovery, auto-slice index, index replication mechanism, restful style interfaces, multiple data sources, such as automatic load search.
- Logstash based on java, is a log collection, filtering, forwarding middleware, is responsible for the various types of log lines of business of the uniform collection, after filtering, forwarding to Elasticsearch for further processing. General work of c / s architecture, client installed on the host side need to collect logs, server side is responsible for each node the received log is filtered, modification and other operations in a concurrent to elasticsearch up.
- Kibana based nodejs, is a visualization tool, but also an open source and free, log analysis Kibana may provide for Logstash and ElasticSearch friendly Web interface, you can help summarize, analyze and search for important data logs.
Elastic Stack technology include the entire department
Data Acquisition Using beats, Filebeat under Beats.
Beats currently contains four tools:
- Packetbeat: collecting network traffic data, the network is a packet analyzer for monitoring the network to collect traffic information, the traffic between the server Package sniffing, parses the application layer information, and the associated message processing, supports ICMP (v4 and v6), DNS, HTTPS, Mysql, Postgresql agreement
- Topbeat: collection systems, processes and file system-level CPU and memory usage data
- Filebeat: collecting data file with field monitoring, mobile server's log files, already replaced Logstash forwarder
- Winlogbeat: collecting Windows event log data
- Mertricbeat: monitoring indicators can regularly obtain information from the external system can be used to monitor the phone Apache, Haproxy, MongoDb, Mysql, Nginx, and other services