Use awk to count the number of visits of a user in nginx

  There will always be attacks online, so you need to analyze the access.log to see that the number of accesses of those users is abnormal. For these abnormal users, you need to deal with them. Take access.log as an example to explain how to count.

The method of counting the number of visits of a user through the access.log log: (The business scenario is login, and then there is only the login url in the access.log, and then we have a specific parameter that needs to be passed by the client. Filter to analyze the behavior, because the data related to the user is encrypted, there is no way to directly filter through the user name, I will process the log content of the access.log, in order not to leak the online data. The uniqueCode below and our log It corresponds to a certain field in the log, but our log is encrypted. For simplicity, no encryption is performed here. The statistical method is the most important.)
1) The contents of the access.log log are as follows:
1.2.3.4 - - [04/May /2018:09:26:09 +0800] "POST /XXX/login.htm HTTP/1.1" 200 896 "0.009" "okhttp/3.9.1" "userName=zhuzi&password=z13y14Bao&uniqueCode=88888888"

2.3.4.5 - - [04/May/2018:09:26:09 +0800] "POST /XXX/login.htm HTTP/1.1" 200 887 "0.010" "okhttp/3.9.1" "userName=zi&password=z13y14Bao&uniqueCode=99999999"

3.2.3.4 - - [04/May/2018:09:26:09 +0800] "POST /XXX/login.htm HTTP/1.1" 200 896 "0.009" "okhttp/3.9.1" "userName=haiyang&password=21131409oui&uniqueCode=11111111"

4.3.4.5 - - [04/May/2018:09:26:09 +0800] "POST /XXX/login.htm HTTP/1.1" 200 887 "0.010" "okhttp/3.9.1" "userName=anya&password=213sdfasdf&uniqueCode=44444444"

4.3.4.5 - - [04/May/2018:09:26:09 +0800] "POST /XXX/login.htm HTTP/1.1" 200 887 "0.010" "okhttp/3.9.1" "userName=anya&password=213sdfasdf&uniqueCode=44444444"

2), take out uniqueCode 

Since uniqueCode is at the end, there is a ", so we can add a &1=1,

sed -i 's/"$/\&1=1"/g' access.log

grep 'uniqueCode' 1_access.log | awk -F '&' '{for(x=1;x<NF;x++)if(index($x, "uniqueCode")>0){print substr($x,12); break}}' >out.log

3), count the number of each appId
cat out.log | awk '{arr[$1]+=1}END{for(i in arr)print i,arr[i]}' | sort -t ' ' - n -k2

 

Alright, let's get to work.

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=325288176&siteId=291194637