1. ELK collects Apache access logs
There are two common ways for ELK to collect logs, namely:
- The format of the source log is not modified, but is filtered and cleaned by the grok method of logstash, and the original irregular log is converted into a regular log.
- Modify the output format of the source log, and output the regular log according to the required log format. Logstash is only responsible for the collection and transmission of logs, without any filtering and cleaning of logs.
These two methods have their own advantages and disadvantages. The first method does not need to modify the original log output format, and directly performs filtering and analysis through the grok method of logstash. The advantage is that it has no impact on the online business system. The disadvantage is that the grok method of logstash is under high pressure. In some cases, it will become a performance bottleneck. If the amount of logs to be analyzed is too large, log filtering analysis may block normal log output. Therefore, when using logstash, if you can use grok, try not to use the grok filtering function.
The disadvantage of the second method is that the output format of the log needs to be defined in advance, which may require a certain amount of work, but the advantage is more obvious, because the required log output format has been defined, and logstash is only responsible for the collection and transmission of logs, which greatly reduces the cost. The burden on logstash is reduced, and logs can be collected and transmitted more efficiently. In addition, currently common web servers, such as apache and nginx, all support custom log output formats. Therefore, in the actual application of enterprises, the second way is the preferred solution.
2. ELK collects Apache access logs
Here we introduce the following architecture:
1. Apache log format and log variables
Apache supports custom output log format, however, apa