After a program is deployed in a test or formal environment, it is often necessary to view logs to help discover and solve problems. For example, it is necessary to count the number of successes and failures of interface calls in the logs to optimize the high concurrency situation.
1. Statistics based on keywords in the log
cat log.txt | grep -o'keyword' | wc -l
Special note : There are many articles on the Internet that directly use cat log.txt | grep'keyword' | wc -l or cat log.txt | grep -c'keyword' is inaccurate, because they are counted by lines, if If all three keywords in the log are on one line, the result will be 1 instead of 3.
Commands used:
grep
The traditional grep program, without parameters, only outputs sentences that match the RE string. Common parameters are as follows:
-v: inverse mode, only output sentences that "do not contain" RE string.
-r: recursive mode, Files in all subdirectories of all levels can be processed at the same time.
-q: Silent mode, no results are output (except stderr. Commonly used to get the return value, if it is true, otherwise it is false.)
-i: Ignore case.
-w: Whole word comparison, similar to \<word\>.
-n: output line numbers at the same time.
-c: output only the number of lines that match the comparison.
-l: output only the file names that match the comparison.
-o: output only the matches RE string. (It is unique to the new version of gnu, not all versions support it.)
-E: Switch to egrep.
wc
parameter:
-c or --bytes or --chars only displays the number of bytes.
-l or --lines only display the number of lines.
-w or --words only display the number of words.
2. Reordering (for neater format)
cat lastb.log | awk '{print $1;$NF}' | sort | uniq -c | sort -nr
Special note : This method is for logs that are more tidy and has more data items, and you can display the counts of all different items at once.
Commands used:
awk
This command has many uses and is more complicated. For detailed understanding, please refer to this https://www.cnblogs.com/-beyond/p/9254007.html , here I will only talk about the ones I used above.
The $1 variable represents the first value of the separation field, and the $NF variable represents the last value of the separation field.
sort
-b Ignore the space characters at the beginning of each line.
-c Check whether the files have been sorted in order.
-d When sorting, deal with English letters, numbers and space characters, and ignore other characters.
-f When sorting, treat lowercase letters as uppercase letters.
-i When sorting, except ASCII characters between 040 and 176, other characters are ignored.
-m Combine several sorted files.
-M Sort the first 3 letters according to the abbreviation of the month.
-n Sort according to the magnitude of the value.
-o<output file> Save the sorted results to the specified file.
-r Sort in reverse order.
-t<separation character> Specify the field separator character used in sorting.
+<Start field>-<End field> Sort by the specified field, and the range is from the start field to the previous field of the end field.
uniq
-c or --count displays the number of times the row repeats next to each column.
-d or --repeated displays only the repeated rows and columns.
-f<field> or --skip-fields=<field> Ignore the fields specified by the comparison.
-s<char position> or --skip-chars=<chars position> Ignore the characters specified by the comparison.
-u or --unique only show the ranks once.
-w<chars position> or --check-chars=<chars position> specifies the characters to be compared.
To be continued. .