overview
This article is based on Honghu version 2.10.0, and aims to provide some references for scenarios where Honghu is used to analyze flush log data. The data used in this article is sample data (not real data). This article mainly uses the flush log to achieve the following analysis:
- Analyze the number of active users
- Analyze the area where users are distributed and display it on the map
- Analyze the user's request and perform further analysis according to the request
- Analyze the situation of the user terminal
Steps
data import
1. Create a data set. If you use an existing data set, you can skip this step
Dataset name: tonghuashun_syslog (the name used in the dashboard, if changed to another, you need to adjust the dataset in the SQL statement after importing the dashboard)
2. Create a data source type
Data source type: tonghuashun_syslog
3. Configure vector.toml. Vector installation reference vector installation ( https://vector.dev/docs/setup/quickstart/#install-vector )
include = ["/data/log/*.json"] # Flush log file path
._datatype = "tonghuashun_syslog" #data source type name
._target_table = "tonghuashun_syslog" #data set name
address = "172.16.1.5:20000" #Honghu IP
field extraction
log analysis
Before field extraction, we need to look at the original log and analyze it
Through the query, you can see that the original log format is as follows (the following example only provides a reference for the log format)
We can roughly judge that the file is a json type file, now to verify
TIPS: How to verify the json format, you can verify it on the online website [Json online verification]
After verification, the json is said to be invalid json, and we try to find a way to complete the json file.
Here we added "{" at the beginning, added "}" at the end, and deleted the "," before "@timestamp", and found that json can be recognized and parsed normally
field extraction
We have verified above that the file can be completed as a json file, then the next step is to determine that the field extraction method is json, and we need to find a way to complete the file as json through the Honghu sql search command. The final sql statement is as follows:
The following is my implementation statement, explain the logic here
The first step: complete the json file
- concat: strings will be concatenated together, this solves the problem of adding "{" and "}"
- substring: returns part of the string input by the first parameter, here I don't need the "," before @timestamp", so "," is the first character in the event, we start from the second character, just Solved the problem of removing ","
Step 2: Use json to parse the completed events
- Above we named the completed content_json, here we parse the completed content_json through parse_json
Step 3: We save our statement as a view "v_tonghuashun", so that we can call the parsed fields later
dash board
Import the Flush Dashboard. New Dashboard > Select Dashboard Profile > OK
The effect is as follows:
Flush—Operation and Maintenance Portrait.json (please go to the official website to obtain the file, and join the "Honghu Technology Exchange Group")
expand
The SQL functions and syntax we have used above and their uses can be queried in the "Help User Manual". For more query and analysis functions, see
https://www.yanhuangdata.com/honghu_manual/docs/search / .
