Drain：An Online Log Parsing Approach with Fixed Depth Tree
Article directory
1 The source of the paper
- Conference/Journal: ICWS (International Conference on Web Services)
- Level: CCF-B
- Time: 2017
2 backgrounds
2.1 Background introduction
Nowadays, more and more developers use existing Web services to build their own systems. In this context, the service management technology based on log analysis, that is, to use service logs to realize automatic or semi-automatic service management, has been obtained. extensive research. Therefore, log analysis techniques that apply data mining models to understand system behavior are widely used in service management.
Most data mining models used in these log analysis techniques require structured input (for example, a list of events or a matrix). However, raw log messages are usually unstructured because developers can write free-text log messages in the source code. Therefore, the first step in log analysis is log parsing, where unstructured log messages are converted into structured events. Unstructured log messages, such as the following example, typically contain various forms of system runtime information: timestamp (recording when the event occurred), verbosity level (indicating the severity level of the event, e.g. INFO), and raw message content (free-text description of a service operation).
2.2 For the problem
With the rapid increase of log volume, model training using offline log parsing methods of all existing logs after log collection becomes very time-consuming, and most existing log parsing methods focus on offline, batch processing of logs superior.
2.3 Innovation points
An online log parsing method for stream processing is proposed, utilizing a deep parsing tree structure.
3 main design ideas
3.1 Drain overall structure
Each path in the parse tree ends with a leaf node which stores a list of log groups, for simplicity we only draw one leaf node here. Each log group has two parts: the log event and the log ID.
- A log event is the template that best describes a log message in the group consisting of the constant parts of the log message.
- log_id will log the ID of the log message in this group.
A special design of parse trees is that all leaf nodes have the same depth, fixed by a predefined parameter depth. For example, the depth of the leaf nodes in Figure 2 is fixed as 3. This parameter limits the number of nodes visited during the search process, greatly improving its efficiency. Furthermore, to avoid branch explosion, we adopted a parameter that limits the maximum number of children of a node.
3.2 Specific steps
(1) Preprocessing according to domain knowledge
Drain allows users to provide simple regular expressions based on domain knowledge representing commonly used variables, such as IP addresses and block IDs. Then, tokens matched by these regular expressions from the raw log message are removed. For example, the block id in Figure 1 will be deleted by "blk[0-9]+". The regular expressions used in this step are usually very simple, and usually only a few of them are needed for a dataset. For example, the data set used in our computation part requires at most two such regular expressions.
(2) Search by log message length
The first-level nodes in the parsing tree represent log groups whose log messages have different log message lengths. In this step, Drain will select a path to the first-level nodes according to the log message length of the preprocessed log messages. For example, for the log message "Receive from node 4", direct the wire to the internal node "Length: 4" in Figure 2. This is based on the assumption that log messages with the same log event are likely to have the same log message length.
(3) Search by the preceding tags
In this step, Drain traverses from a first-level node searched in step 2 to a leaf node. This step is based on the assumption that markers in the beginning of log messages are more likely to be constant. Specifically, Drain selects the next internal node by marking the start position of the log message. For example, for the log message "Receive from node 4", the drain traverses from the first level node "Length: 4" to the second level node "Receive" because the tag at the first position of the log message is "Receive". Drain will then traverse to the leaf node linked to the internal node "Receive" and go to step 4.
(4) Search by tag similarity
In this step, Drain will select the most suitable log group from the list of log groups. We compute the similarity simSeq between log messages and log events for each log group. simSeq is defined as follows:
|
|
Among them, seq1 and seq2 represent log messages and log events respectively; seq (i) is the ith mark of the sequence; n is the length of the log message of the sequence. We compare it with a predefined similarity threshold. If simSeq is much larger than this threshold, Drain will return the group as the most suitable log group. Otherwise, Drain will return a flag (e.g., None in Python) to indicate that there is no suitable log group.
(5) Update the parse tree
If a suitable log group is returned in step 4, the Drain system will add the log ID of the current log message to the log IDs in the returned log group. Additionally, log events in the returned log group will be updated. If Drain cannot find a suitable log group, it will create a new log group based on the current log message, where the log ID only contains the ID of the log message, and the log event is exactly the log message. For example, suppose the current parse tree is the tree on the left in the figure below, and a new log message "Receive 120 bytes" arrives. Drain will then update the parse tree to the tree on the right in the image below. Note that new internal nodes in the third level are coded as "*" because the token "120" contains numbers.
4 Experimental Design
Five log data sets, compared with four traditional methods (2 online and 2 offline), accuracy, precision and recall, and finally anomaly detection comparison based on PCA+ different analysis methods
(1) Accuracy and running time
(2) The impact of different scales on the running time
(3) Abnormal detection results (based on PCA)
5 personal summaries
Drain is an online log parsing method. Regular expressions can be used to separate constants from variables. Improvements can be made to extract parameters and reduce the number of templates. At the same time, this article provides the parsing results of five types of logs and the corresponding setting parameters. Later, you can consider using this method for log parsing, and consider how to introduce parameters for anomaly detection.