First, the background of the demand
of business development increasingly large server more and more
various access logs, application logs, error logs, more and more volume, leading to operation and maintenance personnel can not be good to manage log
developers to troubleshoot a problem and needs to check the log, inconvenience server
operations staff need some data, we need to analyze the operation and maintenance log on the server
Second, why should the ELK
general we need to log analysis scenarios: grep directly in the log file, awk can get they want information want. But in a larger scale is the amount of logs many and complex scenes, this method is inefficient, problems including how to log archiving is too big, too slow how to do text searches, how multi-dimensional queries. Require centralized log management, log on all servers collect aggregate. Common Solutions is creating a centralized log collection system, the logs on all nodes in the unified collection, management, access. Large systems are usually deployed in a distributed architecture, different service modules deployed on a different server, when problems arise, in most cases need to issue critical information exposure, targeted to specific servers and services module, build a centralized logging system that can improve the efficiency of fault location. A complete centralized logging system, need to include the following main features:
Collection - able to collect log data from multiple sources of
transmission - to be able to stabilize the log data to a central system
memory - how to store log data
analysis - analysis supports UI
warning - can provide error reporting, monitoring mechanism
and provides a set of ELK solutions, and is open source software, used in conjunction with each other, the perfect convergence, efficient to meet a number of occasions applications. It is the mainstream one log system.
Three, Introduction ELK
ELK is an abbreviation of three open source software, namely: Elasticsearch, Logstash and Kibana, they are open source software. But now also added a Beats, it is a lightweight log collection processing tools (Agent), Beats small footprint, suitable for transmission to Logstash after collecting log on each server, the government has also recommended this tool, currently due the original ELK Stack members joined the Beats tools so has been renamed Elastic Stack.
Elastic Stack includes:
Elasticsearch is an open source distributed search engine that provides collection, analysis, storage of data three functions. Its features include: distributed, zero-configuration, auto-discovery, auto-slice index, index replication mechanism, restful style interfaces, multiple data sources, such as automatic load search. Details can refer Elasticsearch Definitive Guide
Logstash is mainly used to collect log analysis, log filtering tools to support large amounts of data acquisition mode. General work of c / s architecture, client installed on the host side need to collect logs, server side is responsible for each node the received log is filtered, modification and other operations in a concurrent to elasticsearch up.
Kibana is also an open source and free tools, you can analyze Kibana friendly Web interface and log Logstash ElasticSearch provided to help summarize, analyze and search for important data logs.
Here is a lightweight Beats log collector, in fact, Beats family has six members, early ELK architecture using Logstash collect, parse logs, but Logstash memory, cpu, io and other resource consumption is relatively high. Compared Logstash, Beats share of the system CPU and memory is almost negligible
(after version 5.0) ELK Stack -> Elastic Stack == (ELK Stack + Beats). Beats currently contains six of these tools:
Packetbeat: network data (collect network traffic data)
Metricbeat: Indicators (collection systems, processes and file system-level CPU and memory usage data)
Filebeat: log files (collect file data)
Winlogbeat: Windows Event Log (to collect Windows Event Log data)
Auditbeat: audit data (audit log collection)
Heartbeat: run time monitoring (data collection system run time)
on x-pack tool:
x-pack of Elastic Stack provides security, alarm, monitoring, reporting, charts in one of the expansion pack, for a fee, so this does not involve x-pack installed
four, ELK Chart
This is the simplest way ELK architecture. The advantage is to build a simple, easy to use. The disadvantage is that the larger Logstash consumption of resources, CPU and memory to run high. Further there is no message queue buffer, the presence of hidden data loss. This architecture consists of Logstash collect relevant log distribution, data on each node, and after analysis, it is sent to Elasticsearch filtered on a remote server for storage. Elasticsearch the data in the form of compressed slice store and provide a variety of user API for the query operation. Users can also configure a more intuitive and convenient Kibana Web log queries and generate reports based on the data.
Such architecture introduces message queue mechanism, located Logstash Agent first data on each node / log is passed to Kafka (or the Redis), or data messages in the queue and passed to indirect Logstash, Logstash filtration, transferred to a data analysis after Elasticsearch storage. And finally by Kibana log and data presented to the user. Kafka because of the introduction (or the Redis), even if the distal end Logstash server running stop due to failure, the data will first be stored, to avoid data loss.
Such a framework will collect end logstash replaced beats, more flexible and consume fewer resources, more scalable. At the same time, and can be configured Logstash Elasticsearch cluster to support large cluster system operation and maintenance log data monitoring and queries.
Five, Filebeat works
Filebeat consists of two major components: prospectors and harvesters. These two components work together to change the file sent to the specified output.
Harvester (harvester): responsible for reading the contents of a single file. Each file will start a Harvester, Harvester for each individual file will be read line by line, send the file contents to the development of the output. Harvester is responsible for opening and closing files, which means when Harvester run, the file descriptor is open, if the file is renamed or deleted in the collection, Filebeat will continue to read this file. So before Harvester closed, the disk will not be released. By default filebeat will keep the file open, until it reaches close_inactive (if this option is turned on, filebeat will no longer update the file handle is closed within the specified time, the time to read the time from the last row harvester starts. If the file handle is after closing, the file is changed, it will start a new harvester. Close the file handle time does not depend on the file modification time, if this parameter is configured incorrectly, the situation is not real-time log that may occur, is determined by scan_frequency parameters, default 10s .Harvester using the internal time stamp to record the file was last collected, for example: set 5m, then after the last line of the file read Harvester, countdown starts five minutes, within 5 minutes if no change file, close the file handle default. 5m).
Prospector (Surveyor): responsible for managing Harvester and find all read the source.
filebeat.prospectors:
- input_type: log
Paths:
- /apps/logs/*/info.log
Prospector will find / apps / logs / * info.log all files in a directory, and start a Harvester for each file. Prospector checks each file to see whether the Harvester has been launched, the need to start, or if the file can be ignored. If Harvester closed, only Prospector check will be performed in the event of changes in file size. Only detect the local paper.
How Filebeat record file status:
The state file records in the file (default / var / lib / filebeat / registry). This status can remember offset Harvester collect documents. If the connection is not output devices, such as ES like, filebeat records the last row before the transmission time can be connected and then continue to send. Filebeat at run time, Prospector state will be recorded in memory. Filebeat restart time, using the state registry records to reconstruct, to restore to the state before the restart. Each Prospector will record a status for each file found, for each file, Filebeat whether the unique identifier stored previously collected to detect file.
How Filebeat ensure the event is output at least once:
Filebeat has been able to ensure the event is passed to the output configuration at least once, no data is lost, because filebeat the delivery status of each event is saved in a file. When the output side has not been confirmed, filebeat attempts have been sent until get a response. If filebeat be closed during transmission, it will not confirm that all events before then close. Before filebeat close to any confirmation of the time, it will be re-sent after filebeat restart. This ensures that at least once sent, but there may be repeated. Shutdown_timeout by setting parameters to set off before waiting for an event response time (disabled by default).
Six, Logstash works
Logstash event handler has three stages: inputs → filters → outputs. It is a receiving, processing, forwarding logging tool. Support system log, webserver logs, error logs, application logs, in short, can throw to include all types of logs.
Input: Input data into logstash.
Some common inputs:
File: read from the file system, similar to the tail -f command
syslog: monitoring system log messages on port 514, and analyzes in accordance with the standard RFC3164
redis: redis service is read from the
beats: the reading from filebeat
Filters: intermediate processing data, operate on the data.
Some common filters are:
Grok: parse any text data, Grok is the most important plug-Logstash. Its main role is to string text format, converted into a specific structured data, with the use of regular expressions. Built more than 120 parsing grammar. Grok official expression of: https: //github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns,grok online debugging: https: //grokdebug.herokuapp.com/. Also on kibana also provides grok debugger.
mutate: the conversion of the field. For example, the field delete, replace, modify, rename, and so on.
drop: not discard a portion of the processing events.
clone: copy event, the process may add or remove fields.
geoip: add geographic information (for the front kibana graphical display use)
Outputs: outputs is most tip assembly logstash processing pipeline.
An event can go through multiple outputs in the process, but once all the outputs are executed end, this event will complete the life cycle.
Some common outputs are:
elasticsearch: data can be efficiently stored and can be queried convenient and simple.
file: Save the event data to a file.
graphite: transmitting event data to the graphic component assembly, a very popular open source store graphical display.
Codecs: codecs are based on data flow filter, it can be used as input, output part of the configuration.
Codecs can help you easily have been serialized data sent over the split.
Some common codecs:
json: data encoding / decoding using json format.
multiline: The exchange of data aggregated into multiple events in a single row. For example: java exception information and stack information.
Logstash example:
Log Format
2019-07-04 14:07:19.916 INFO 7 --- [o-9090-exec-843]o.a.imocker.controller.ApiController: [ApiMockRequest]api:/brScorecashon/nlfc/jsonTonlfcService.do, method:POST
自定义pattern
LOG_TIMESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})
LOG_LEVEL ERROR|DEBUG| INFO| WARN
LOG_THREAD \[.*?\]
LOG_MSG_SPLIT [\s]*:
LOG_MSG [\s\S]*
logstash.conf
input {
beats {
port => 5044
client_inactivity_timeout => 36000
}
}
filter {
grok {
patterns_dir => "/usr/share/logstash/config/patterns"
match => {"message" => "%{LOG_TIMESTAMP:log_date} %{LOG_LEVEL:log_level} %{INT:pid} --- %{LOG_THREAD:log_thread} %{JAVACLASS:java_class} %{LOG_MSG_SPLIT} %{LOG_MSG:log_msg}"}
}
date {
match => ["log_date", "yyyy-MM-dd HH:mm:ss.SSS"]
target => "@timestamp"
timezone => "+08:00"
}
mutate {
strip => ["log_level"]
rename => {"[host][name]" => "host"}
}
}
output {
elasticsearch {
= the hosts> [ "http://192.168.66.41:9201","http://192.168.66.41:9202","http://192.168.66.41:9203"]
index => "imocker - {+ YYYY% } .MM.dd "
}
stdout {
CODEC => rubydebug
}
}
seven, Kibana
Kibana is an open source analysis and visualization platform, designed to work together and Elasticsearch. You can use Kibana to search, view, and Elasticsearch and the data stored in the index to interact. You can easily perform advanced data analysis, and icons in various forms, tables, and maps of visual data. Kibana appreciated that such a large amount of data becomes easy. It's simple, browser-based interface allows you to quickly create and share dynamic dashboards, real-time display changes Elasticsearch query. Kibana is a Web application, you can access it by 5601. For example: http://192.168.66.41:5601. Before we officially use Kibana, need to match our Elasticsearch the index database, because our Elasticsearch there may be a lot of index database, Kibana to performance factors, we will not advance all the indexes libraries guide come in, we need with the index on which index guide. According to the following steps: Management >> Index Patterns >> Create Index Patterns then we can see the following interface:
In the index pattern index database input box, you can use fuzzy matching query, for example, we are ready to import imocker log data, enter "imocker- *", when a match is found, below the input box will be a successful prompt, and the right Next step will come out a button, click the button to enter the next step, and then click Create index pattern matching is completed:
Then Kibana time we need to select a field, because of what these three indexes are in accordance with time-correlated, we choose the time field is @timestamp.
So, now we have to match these three types of test data to Kibana in, and then you can use the Kibana.
Discover
You can explore your data from the Discover page interactive. You can access each index with the index of the selected default matching each document. You can submit queries, search filter structure, and view the document data. You can also see the number of documents that match the query request, and the field value statistics. If you choose to configure the time index mode field, the document distribution over time will be displayed in the histogram at the top of the page.
Script Fields
ALL
Monitoring alarm
Kaae, sentinl, elastalert
----------------
Disclaimer: This article is the original article CSDN bloggers' cup of rum ", and follow CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source and link this statement.
Original link: https: //blog.csdn.net/u012562943/article/details/101060053