logagent is a deamon process deployed on each server, which acts as a transmission channel for log data. Provide HTTP service to pull logs, the structure is as follows: 
Among them,
- filter: request filter, responsible for access source analysis statistics, authorization authentication, etc.
- container: contains the set of file handles being accessed, which can prevent wrong positioning when log rolling file name changes occur
- handler: The handler for obtaining log data, using the method of random reading of offsets
- storer: saves the file location information successfully obtained by the user
1. Long Links vs Short Links
http1.1 supports long links by default to avoid performance consumption caused by repeated connection establishment. However, long links will occupy system resources, and there will be hidden dangers when a large number of requests are made. Therefore, the short link method is selected, that is , the keep-alive in http is turned off.
2. Request Specification
http://127.0.0.1:8080/get?file=/home/admin/tomcat/logs/catalina.log&user=test&pwd=123456
in,
get : indicates that it is a request to obtain log data
file : the full path of the log file, supports wildcards * to specify all log files in a directory
user : the requesting user
pwd : the authorization key of the requesting user, encrypted with md53. Stability
The logagent is deployed together with the business, and the following restrictions are made considering the stability
- The maximum number of files that a single user can access at the same time on an online physical machine is 100
- The maximum number of links allowed at the same time on an online physical machine is 200
- The maximum size of an online physical machine that allows a single user to pull logs at a time is 20M
When the speed of pulling logs cannot keep up with the speed of logging, it is necessary to increase the limit size or concurrency
If the logagent is accidentally down , it supports to continue to obtain data from the last location after recovery
4. Performance plan
Because agnt is written in java, it mainly considers the impact of the process gc on the host business system under full load operation. The specific test shall prevail.
5. Security Policy
- Permission to access files
- Authenticate user and authorization password in request
- The whitelist authorization of the collection machine connected to the agent
6. Log module design
- Encoding: uniformly use utf-8
- Content interception
Ordinary log files are records separated by newline characters. When intercepting, a complete log should be obtained, that is,
Current site + single maximum supported transfer amount = newline, intercepted as normal
The current position + the maximum supported transfer amount in a single time ! = newline, go forward to find the latest newline to intercept
Current location + single maximum supported transfer amount > file end , directly intercepted to file end
Extended consideration to support user-specified delimiters for cutting
- site access
According to the location information of the file stored by the user, a file can support multiple users to pull data at the same time, that is, the file will retain the location information of these users respectively. The retention path is specified by logagent by default, and the strategy of asynchronous update after successful pulling is adopted.
- Support log rolling
When the log file is rolled, it is possible to locate the old file that has not been pulled. Keep the file handle ( inode ), get the current file size every time you pull data, until the handle file content is pulled
7. Operation and maintenance support
- Permissions: including machines, files, whitelists
- Provide basic services such as obtaining all ip addresses of an application
- Installation Deployment & Remote Control
The attachment focus.zip is the logagent source code provided for reference~