Awk is a very useful text processing tool on Linux. Compared with the processing of a whole line of sed commands, awk is often used for column processing, such as printing the third column of a specified line. This article describes the use of the awk command through common examples of the awk command.
awk basic syntax
awk [-F delimiter] 'Condition Type 1{Action 1} Condition Type 2{Action 2}...' input-file
illustrate:
(1) -F means to set the separator, the default is a space. The condition type indicates the matching condition for each line, which can be a conditional statement, a compound statement or a regular expression. The action indicates the command to be executed. The command must be enclosed in curly brackets {}
(2) Any awk statement consists of patterns and actions.
(3) Whether the operation command is executed is related to whether the matching condition is matched. If it matches, the corresponding command will be executed, if not, it will not be executed.
(4) Whether the mode is set is optional. If mode is not set, the command is executed for all each line.
(5) $0 means row, $1, $2, $3...$n means column 1, column 2, column 3... column n respectively.
awk processing flow:
(1) Read in the first line, and fill in the data of the first line into variables such as $0, $1, $2, and $3;
(2) According to the restriction of the condition type, it is judged whether the following action needs to be performed. until all actions and condition types are done.
(3) Repeat steps (1) and (2) for each line until all lines of the file are processed.
A simple example, assuming the content of the servers.txt file is as follows
1:192.168.0.1:root:root 2:192.168.0.2:oracle:oracle 3:192.168.0.3:admin:admin
Print the contents of columns 2, 3, and 4 in the file. The awk command is as follows:
awk -F ':' '{print $2,$3,$4}' services.txt
Description: -F ':' means to use ":" colon as separator, print $2, $3, $4 means to print column 2/3/4. Set multiple delimiters if needed. You can use -F '[delimiter 1 delimiter 2...]', such as -F '[:/]' means to use: colon and / backslash as delimiters at the same time.
Regular expressions and conditional operators in awk
The characters often used in regular expression matching operations in awk are as follows:
. * + ? $ | \ [] {} ()
The conditional operators are as follows:
<(less than) <=(less than or equal to) ==(equal to) >(greater than) >=(greater than or equal to) !=(not equal to) ~(matches regular expression) !~(does not match regular expression)
Among them, the usage of operators such as <, >, and <= is relatively intuitive. Here, we mainly talk about the usage of two expressions, ~ (matching regular expressions) and !~ (not matching regular expressions).
The basic example is as follows:
(1) Print the line where the first field contains the root string
awk -F":" '{ if ($1~/root/) print $0}' /etc/passwd #Writing one
awk -F ":" '$1~/root/{ print $0}' /etc/passwd #Writing two
(2) Print the line where the first field does not contain the root string
awk -F":" '{ if ($1 !~ /root/) print $0}' /etc/passwd #Writing one
awk -F ":" '$1 !~ /root/{ print $0}' /etc/passwd #Writing two
(3) Print the line where the first field is equal to the root string
awk -F":" '{ if ($1 == 'root') print $0}' /etc/passwd #Writing one
awk -F ":" '$1 == 'root'{ print $0}' /etc/passwd #Writing two
(4) Print the line where the first field is not equal to the root string
awk -F":" '{ if ($1 != 'root') print $0}' /etc/passwd #Writing one
awk -F ":" '$1 != 'root'{ print $0}' /etc/passwd #Writing two
(5) Print users whose user ID is less than 500
awk -F":" '{if($2<500)print $0}' /etc/passwd
(6) Print the line whose line header is oracle
awk -F":" '{if($1~/^oracle/)print $0}' /etc/passwd
(7) Print the line matching the oracle or root string
awk -F":" '{ if ($1~/(root|oracle)/) print $0}' /etc/passwd #Use regular expression metacharacter |
awk -F ":" '{ if ($1~/root/ || $1~/oracle/) print $0}' /etc/passwd #Use or ||
Description: The vertical bar "|" means to match one of the patterns on both sides of the vertical bar "|". When using the vertical bar, the statement must be enclosed in parentheses ().
(8) Print lines that match both oracle and root
awk '{if($1=='root' && $2=='500')print $0}' /etc/passwd
awk built-in variables
The built-in variables of awk include FS, NF, OFS, ORS, NR, etc. The meanings and usages are listed below.
(1) FS indicates the separator used to set the column, which has the same function as the -F option. Space by default. An example is as follows:
echo "2017-12-20" |awk 'BEGIN{FS="-"}{ print $1,$2,$3 }' #Output 2017 12 20
(2) NF represents the total number of columns in a row of records. If you need to print the content of the last column, for example:
echo "/usr/local/bin/python" | awk -F '/' '{ print $NF}' #output python
Description: NF is often used to print the contents of the column specified by the reciprocal.
(3) OFS indicates the specified output field separator, the default is a space. An example is as follows:
echo "1990 09 09" | awk 'BEGIN{OFS="-"}{ print $1,$2,$3}' #output 1990-09-09
(4) ORS represents the separator of the output line record, the default is the newline character (\n). If you need to set the delimiter of the output line to #, it can be expressed as follows:
echo -e "1990\n09\n09" | awk 'BEGIN{ORS="#"}{ print $0 }' #output 1990#09#09#
(5) NR represents the number of records that have been read. If you need to print the number of lines in the file, an example is as follows:
awk 'END{print NR}' /etc/passwd
awk built-in string functions
(1) length(string) means to calculate the length of the string string, and the return value is the length of the string. An example is as follows:
echo "Hello World" | awk '{print length($0)}' #输出11
(2) index(string, substr) indicates the position of the character substr to be searched in the string string, where substr must be enclosed in double quotation marks. An example is as follows:
echo "Hello World" | awk '{print index($0,"o")}' #输出5 awk 'BEGIN{print index("Hello World","o")}' #输出5
(3) match(string,regex) Tests whether the string string contains a string that matches the regular expression regex, and the return value is the number of characters that appear successfully. If it is not found, it returns 0. An example is as follows:
awk 'BEGIN{ print match("Hello World",'o')}' #Find the first occurrence of the string o, output 1
echo "Hello World" | awk '{ print match($0,'o') }' #Same as above, output 1
awk 'BEGIN{ print match("Hello World",/[Oo]/)}' #Find the first occurrence of uppercase or lowercase o in the string. output 5
(4) split(string,arr,fs) Indicates that the specified separator fs is used to separate the string string, and the result of the split is saved to the arr array. The split() function returns the length of the array. An example is as follows:
echo "123#456#789" | awk '{len=split($0,arr,"#");for(i=1;i<=len;i++)print arr[i]}'
Description: Through the obtained array length, the content in the array is obtained through a loop, where the array subscript is calculated from 1.
(5) sub(regex,string) means to replace the content matched by the regular expression regex with the string string. An example is as follows:
echo "Java:Python:Java:C" | awk 'sub(/Java/,"Ruby",$0)' #输出Ruby:Python:Java:C
(6) substr(string,pos) Get the content after the specified position pos in the string sting. Examples are as follows
echo "Java:Python:C" | awk '{ print substr($0,1,4)}' #output Java
(7) substr(string,pos,n) Get the content of length n from the specified position pos in the string string
echo "Java:Python:C" | awk '{ print substr($0,6)}' #Output Python :C
(8) gsub(regex,string) means to use the string string to replace the content matched by the regular expression regex. An example is as follows:
echo "Java:Python:Java:C" | awk 'gsub(/Java/,"Ruby")' #输出Ruby:Python:Ruby:C
(9) gsub(regex,string,target) Indicates that within the specified range, use the string string to replace the content matched by the regular expression regex. An example is as follows:
echo "Learn Shell at 2013-08-08" | awk 'gsub(/-/,":",$4)' #输出Learn Shell at 2013:08:08
Summarize
This article introduces the basic syntax and common examples of the awk command. Awk has powerful text processing capabilities, and mastering the basic content of awk commands can complete most of the problems encountered in work. In addition, awk is very efficient in constructing large batches of test data.