According to the position where the extraction rules are run, there are two types of practices, one is at the center side and the other is at the log side.
The central end is to transmit the logs of all the machines to be processed to the central center, such as through Kafka transmission, and finally fall to Elasticsearch. The index extraction rules can be inserted into the Kafka channel as a stream computing task, and the performance and real-time performance are relatively better. Or directly write a scheduled task, call the interface of Elasticsearch to query the log, and at the same time provide the aggregation calculation function, let Elasticsearch return the indicator data, and then write it into the timing library. The real-time performance will be poor, but it is basically enough.
Log-side processing means that the extraction rules run directly on the machine that generates the log, read the log in a streaming manner, and match the regular expression. For the hit log, it is sometimes valuable to extract the number part of it as an indicator, or not to extract any number, and only count the number of hit log lines. For example, count the number of occurrences of the Error or Exception keyword, and we know the system Is it an error?
There are many open source solutions for the log terminal processing method, the more famous ones are mtail and grok_exporter.
Tools such as mtail and grok_exporter are like executing tail -f on a log file, and then each time a log is received, it will match a predefined regular expression. If the match is successful, it will perform certain actions, otherwise it will skip waiting for the next log .
For the deployment of mtail, if there are 5 applications on a machine that need to use mtail to extract indicators, and the log formats of each application are different, it is recommended to start 5 mtail processes to process them separately. Although it is troublesome to manage, the performance is good and there is no influence on each other.
If you put all the extraction rules in one directory, and then specify the log paths of these multiple applications at the same time through the -logs parameter multiple times, one mtail process can also handle it, but for each line of logs, mtail will take all the extraction rules It is a waste of performance to run them all once, and the speed of regular extraction is not fast. In addition, some indicators may be reused by all applications. If they are processed together, they will easily interfere with each other, resulting in inaccurate statistical data. From these two points of view, try to disassemble and deal with them separately. Although it is more troublesome to manage, it is worth it.
There is no such problem in the container scenario. The container scenario can be deployed directly using sidecar. Each Pod must have only one application, and the associated mtail can only focus on processing the logs of this application.
There are several ways to extract indicators, generally speaking, there are two types: the central terminal and the log terminal. Since the processing methods of the central terminal are more common in commercial software, no open source solutions have been seen. The core processing logic of the log side is the same. The log content is continuously read through a method similar to tail -f, and then regular matching is performed on each line of log. Since the log format is not fixed, it is difficult to have a structured processing method, so these The tools all choose to use regular methods to extract filter indicators.
This article is a study note for Day13 in August. The content comes from Geek Time "Operation and Maintenance Monitoring System Practical Notes". This course is recommended.