Common commands for linux log analysis

1. Log files

java.lang.ArrayIndexOutOfBoundsException: 5
	at com.xxx.ds.bo.BO.setEntity(Unknown Source)	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.ArrayIndexOutOfBoundsException: 7
	at com.xxx.ds.bo.BO.setEntity(Unknown Source)	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

java.lang.NullPointerException
	at com.xxx.ds.dao.DaoImpl.addTrail(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)
java.lang.ArrayIndexOutOfBoundsException: 10
	at com.xxx.ds.bo.BO.setEntity(Unknown Source)	
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.NullPointerException
	at com.xxx.ds.dao.DaoImpl.addTrail(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)

2. Log deduplication

# cat syslog_1.log syslog_2.log|sed '/^\s*$/d' |sort |uniq >syslog_uniq.log

	at com.xxx.ds.bo.BO.setEntity(Unknown Source)	
	at com.xxx.ds.dao.DaoImpl.addTrail(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.ArrayIndexOutOfBoundsException: 10
java.lang.ArrayIndexOutOfBoundsException: 5
java.lang.ArrayIndexOutOfBoundsException: 7
java.lang.NullPointerException

3. Log filtering

# cat syslog_uniq.log |grep -v "ArrayIndexOutOfBoundsException:" >syslog_grep-v.log

	at com.xxx.ds.bo.BO.setEntity(Unknown Source)	
	at com.xxx.ds.dao.DaoImpl.addTrail(Unknown Source)
	at java.lang.Thread.run(Thread.java:744)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.NullPointerException

-v: Indicates that the content in the regular expression is not included

-E: multiple keywords ( grep -E "word1|word2|word3" file.txt)

4. Command description

4.1 uniq

Usage: uniq [options]... [files]

Filter adjacent matching lines from the input file or standard input and write to the output file or standard output. Matching lines will be merged at their first occurrence when no options are appended.

Parameters that are required for long options are also required for short options.

-c, --count //Prefix each line with a prefix number indicating the number of occurrences of the corresponding line  
 -d, --repeated // only output repeated lines  
 -D, --all-repeated //Only output repeated lines, but a few lines output a few lines  
 -f, --skip-fields=N //-f ignores the number of fields, -f 1 ignores the first field  
 -i, --ignore-case // case insensitive  
 -s, --skip-chars=N //The root -f is a bit like, but -s is ignored, how many characters after -s 5 ignore the last 5 characters  
 -u, --unique //After removing duplicates, all are displayed. The distinct function of root mysql is a bit like  
 -z, --zero-terminated   //end lines with 0 byte, not newline  
 -w, --check-chars=N //Do not compare the content after the Nth character of each line  
 --help //Display this help message and exit  
 --version //Display version information and exit

For a detailed explanation of the command, see: http://blog.51yip.com/shell/1022.html

4.2 Filter blank lines

The grep, sed, awk and tr commands can all delete blank lines in a file.

4.2.1 grep

# grep . data.txt //The efficiency is relatively high

# grep -v '^$' data.txt

# grep '[^$]' data.txt

4.2.2 sed

# sed '/^$/d' data.txt

# sed '/^\s*$/d' data.txt //The command can also delete blank lines composed of complete spaces, tabs, etc.

4.2.3 awk

# awk NF data.txt //The command can also delete blank lines composed of spaces, tabs, etc.

# awk '!/^$/' data.txt

awk concise tutorial, see: http://coolshell.cn/articles/9070.html

4.2.4 tr

tr -s '\n' < data.txt

Common commands for linux log analysis

Guess you like