1. Basic grammar
awk '{pattern + action}' {filenames}
In awk syntax, pattern represents what to look for in the data, and action is a series of commands that are executed when a match is found. Curly braces ({}) do not need to appear all the time in a program, but they are used to group a sequence of instructions according to a specific pattern. pattern is the regular expression to be represented, enclosed in slashes.
2, awk principle
awk executes this script for each line in the input file:
$ awk -F":" '{ print $1 }' /etc/passwd
$ awk -F":" '{ print $1 $3 }' /etc/passwd
$ awk -F":" '{ print $1 " " $3 }' /etc/passwd
$ awk -F":" '{ print "username: " $1 "\t\tuid:" $3" }' /etc/passwd
-F parameter: Specify the delimiter, you can specify one or more
prints followed by string concatenation
NF: The number of fields separated by delimiters in a record
NR: The number of records that have been read, which is the line number, starting from 1
Example 1: Only view the content of lines 20 to 30 in the test.txt file (100 lines)
[root@local]# awk '{if(NR>=5 && NR<=10) print $1}' test.txt
5
6
7
8
9
10
Example 2: It is known that the content of the test.txt file is
[root@local]# cat test.txt I am Poe,my qq is 168
Please filter out the 'Poe' string and 168 from this file, and the final output is: Poe 168
[root@localhost]# awk -F '[ ,]+' '{print $3" "$7}' test.txt
Poe 168
3. BEGIN and END modules
Typically, awk executes each block of script code once for each input line. However, in many programming situations, initialization code may need to be executed before awk begins processing the text in the input file. For this case, awk allows you to define a BEGIN block.
Because awk executes the BEGIN block before it starts processing the input file, it is an excellent place to initialize FS (field separator) variables, print headers, or initialize other global variables that will be referenced later in the program.
awk also provides another special block, called the END block. awk executes this block after processing all lines in the input file. Typically, END blocks are used to perform final computations or to print summary information that should appear at the end of the output stream.
Example 1: Count the number of accounts in /etc/passwd
[root@local]# awk '{count++;print $0;} END{print "user count is ",count}' ／etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
………………………………………………………………………………
www:x:80:80::/home/www:/bin/bash
user count is 25
count is a custom variable. There is only one print in the previous action{}, in fact, the print is just a statement, and the action{} can have multiple statements, separated by ;. There is no initialization of count here. Although the default is 0, the proper way is to initialize it to 0:
[root@local]# awk 'BEGIN {count=0;print "[start] user count is ",count} {count=count+1;print $0} END{print "[end] user count is ",count}' /etc/passwd
[start] user count is 0
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
………………………………………………………………………………
www:x:80:80::/home/www:/bin/bash
[end] user count is 25
Example 2: Count the number of bytes occupied by files in a folder
[root@local]# ll |awk 'BEGIN {size=0;} {size=size+$5;} END{print "[end]size is ",size}'
[end]size is 24692
If displayed in MB:
[root@local]# ll |awk 'BEGIN{size=0;} {size=size+$5;} END{print "[end]size is ",size/1024/1024,"MB"}'
[end]size is 0.0235481 MB
4, awk operator
awk assignment operator: a+5; equivalent to: a=a+5; other similar
[root@local]# awk 'BEGIN{a=5;a+=5;print a}'
10
awk logical operator: 0 represents False in the output, 1 represents True
[root@test03 opt]# awk 'BEGIN {a=1;b=2;print (a>2 && b>1,a=1 || b>1)}'
0 1
awk regular operator: if the value assigned by a contains 100, the output is ok
[root@Gin scripts]# awk 'BEGIN{a="100testaa";if(a~/100/) {print "ok"}}'
ok
[root@local]# echo | awk 'BEGIN{a="test100"}a~/test/{print "ok"}'
ok
Relational operators:Such as: > < can be used as a string comparison, can also be used as a numerical comparison, the key is that if the operand is a string, it will be converted to a string comparison. Both are numbers before they are converted to numerical comparisons. String comparison: Compare in ascii code order.
[root@local]# awk 'BEGIN{a="11";if(a>=9){print "ok"}}' #无输出
[root@local]# awk 'BEGIN{a=11;if(a>=9){print "ok"}}'
ok
[root@local awk 'BEGIN{a;if(a>=b){print "ok"}}'
ok
awk arithmetic operators:Description, all operations used as arithmetic operators, the operands are automatically converted to numeric values, and all non-numeric values become 0.
[root@Gin scripts]# awk 'BEGIN{a="b";print a=="b"?"ok":"err"}'
ok
[root@Gin scripts]# awk 'BEGIN{a="b";print a=="c"?"ok":"err"}'
err
5. Commonly used awk built-in variables
Note: There are many built-in variables, see related information
field separator FS
FS="\t" one or more tabs separated
[root@local]# cat test.txt
ww CC IDD
[root@local]# awk 'BEGIN{FS="\t+"}{print $1,$2,$3}' test.txt
ww CC IDD
FS="[[:space:]+]" one or more blank spaces, the default
[root@local]# cat test.txt
we are studing awk now!
[root@local]# awk -F [[:space:]+] '{print $1,$2,$3,$4,$5}' test.txt
we are
[root@local]# awk -F [[:space:]+] '{print $1,$2}' test.txt
we are
FS="[" ":]+" separated by one or more spaces or :
[root@local]# cat test.txt
root:x:0:0:root:/root:/bin/bash
[root@local]# awk -F [" ":]+ '{print $1,$2,$3}' test.txt
root x 0
Number of fields NF
[root@local]# cat test.txt
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin:888
[root@local]# awk -F ":" 'NF==8{print $0}' test.txt
bin:x:1:1:bin:/bin:/sbin/nologin:888
Number of records NR
[root@local]# ifconfig eth0|awk -F [" ":]+ 'NR==2{print $4}' ## NR==2 is to take the second line
192.168.168.10
RS record separator variableSetting FS to "\n" tells awk that each field occupies one line. By setting RS to "", you also tell awk that each address record is separated by a blank line.
[root@local]# cat area.txt
Jimmy the Weasel
100 Pleasant Drive
San Francisco,CA 123456
Big Tony
200 Incognito Ave.
Suburbia, WA 64890
[root@local]# cat awk.txt
#!/bin/awk
BEGIN {
FS="\n"
RS=""
}
{
print $1","$2","$3
}
[root@local]# awk -f awk.txt area.txt
Jimmy the Weasel,100 Pleasant Drive,San Francisco,CA 123456
Big Tony,200 Incognito Ave.,Suburbia,WA 64890
OFS output field separator
[root@local]# cat hello.txt
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin:888
[root@local]# awk 'BEGIN{FS=":"}{print $1","$2","$3}' hello.txt
root,x,0
am,x,1
[root@local]# awk 'BEGIN{FS=":";OFS="#"}{print $1,$2,$3}' hello.txt
root#x#0
bin#x#1
ORS output record separator
[root@local]# cat area.txt
Jimmy the Weasel
100 Pleasant Drive
San Francisco,CA 123456
Big Tony
200 Incognito Ave.
Suburbia, WA 64890
[root@Gin scripts]# cat awk.txt
#!/bin/awk
BEGIN {
FS="\n"
RS=""
ORS="\n\n"
}
{
print $1","$2","$3
}
[root@local]# awk -f awk.txt area.txt
Jimmy the Weasel,100 Pleasant Drive,San Francisco,CA 123456
Big Tony,200 Incognito Ave.,Suburbia,WA 64890
6, awk regular
Regular application of
regular expressions
awk '/REG/{action}' file, /REG/ is a regular expression, you can send records that meet the conditions in $0 to: action for processing
[root@local]# awk '/root/{print $0}' passwd ##match all lines containing root
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
[root@local]# awk -F: '$5~/root/{print $0}' passwd ## 以分号作为分隔符，匹配第5个字段是root的行
root:x:0:0:root:/root:/bin/bash
[root@local]# ifconfig eth0|awk 'BEGIN{FS="[[:space:]:]+"} NR==2{print $4}'
192.168.168.10
布尔表达式
awk '布尔表达式{action}' file仅当对前面的布尔表达式求值为真时，awk才执行代码块。
[root@local]# awk -F: '$1=="root"{print $0}' /etc/passwd
root:x:0:0:root:/root:/bin/bash
[root@local]# awk -F: '($1=="root")&&($5=="root") {print $0}' /etc/passwd
root:x:0:0:root:/root:/bin/bash