Intelligent operation and maintenance (AiOps) is developed on the basis of automated operation and maintenance (DevOps). It can perform system log and application log collection, analysis, display and other comprehensive log management solutions for automated operation and maintenance at the application level. Program.
Below we will analyze the following solutions that we often use, so that we can choose a log management solution suitable for our company in actual work and provide good log data services for company operations.
l ANY
ELK is the abbreviation of ElasticSearch , Logstash , and Kibana , which provide search, data access and visualization functions respectively, and constitute the application technology stack of Elastic .
ü Elastic introduction
ElasticSearch is an open source search service based on Lucene . It provides a full-text search engine with distributed multi-user capabilities based on a RESTful web interface.
Elasticsearch is developed in Java and released as an open source under the terms of the Apache license. It is a popular enterprise search engine. Designed for use in cloud computing, it can achieve real-time search, stable, reliable, fast, and easy to install and use.
ElasticSearch provides REST API for cluster management, monitoring, and health check.
Elasticsearch is a quasi-real-time search platform. There is a certain delay from data indexing to data being searched.
ü Introduction to Logstash
Logstash is implemented in jruby , similar to a data pipeline, processing, deforming, filtering the input data, and then outputting it to other places.
Logstash has designed its own DSL , including areas, comments, data types ( boolean, string, numeric, array, hash ) , conditional judgment, field references, etc.
Logstash 's data pipeline consists of three steps, Input , Filter and Output , and each step can be extended by plugin . In addition, Input and Output also support the configuration of Codecs to complete the coding and decoding of input and output data.
Logstash support of common Input contains File , syslog , Beats and so on. Filter mainly completes the formatting of the data, you can add, delete, change fields, add tags, etc.
Logstash in Output plug not only support elasticsearch , and many other software can also be integrated and goals, Output can be a file, database, S3 , Hadoop , and so on.
In actual use, the logstash process will be divided into two different roles. Run on the application server, try to reduce the operating pressure, only read and forward, this role is called shipper ; run on a separate server, complete data analysis processing, responsible for writing to Elasticsearch role, called indexer .
As a stateless software, logstash can easily scale linearly with the message queuing system
Events Discounts Forum Login Register Kibana Kai绍
Kibana is an open source analysis and visualization platform for Elasticsearch , used to search, view, and interact with data stored in Elasticsearch indexes. Use Kibana to perform advanced data analysis and display through various charts.
l Splunk
Splunk is the first company listed on Nasdaq in the field of big data.
Splunk provides a search engine for machine data. Use Splunk to collect, index, and utilize fast-moving computer data generated by all applications, servers, and devices (physical, virtual machines, and cloud platforms) .
Splunk processes data quickly, allowing you to find and solve problems within minutes, and it can correlate and analyze complex events that span multiple systems. In order to get more operational visibility forecasts and business intelligence reminders.