Detailed awk command

awk command

• Awk is also a streaming editor, operating on lines and segments in a document. Awk can be divided into several parts:
• match character or string
• Cut out a section of a document
• Conditional operator
• computation
• built-in variable

Example 1:

• head -n2 test.txt|awk -F ':' '{print $1}'
• head -n2 test.txt|awk -F ':' '{print $0}'
• awk -F ':' '{print $1"#"$2"#"$3"#"$4}'
• awk '/oo/' test.txt
• awk -F ':' '$1 ~/oo/' test.txt
• awk -F ':' '/root/ {print $1,$3} /test/ {print $1,$3}' test.txt
• awk -F ':' '$3=="0"' /etc/passwd
• awk -F ':' '$3>="500"' /etc/passwd
• awk -F ':' '$3>=500' /etc/passwd
• awk -F ':' '$7!="/sbin/nologin"' /etc/passwd

match character or string

1. line that matches root

[root@linux-128 ~]# awk '/root/' 1.txt
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

2 awk can match a character in a certain paragraph, separated by: to match the line with root in the first paragraph

[root@linux-128 ~]# awk -F ':' '$1 ~ /root/' 1.txt
root:x:0:0:root:/root:/bin/bash

1. awk matching characters can use regular

[root@linux-128 ~]# awk '/ooo?/' 1.txt
root:x:0:0:root:/root:/bin/bash
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin

1. awk has its own free function, grep free needs -E or egrep, sed free needs -r option

[root@linux-128 ~]# awk -F ':' '/oo+/' 1.txt
root:x:0:0:root:/root:/bin/bash
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin

grep -E

[root@linux-128 ~]# grep -E 'oo+' 1.txt
root:x:0:0:root:/root:/bin/bash
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin

but –r

[root@linux-128 ~]# sed -n -r '/oo+/'p 1.txt
root:x:0:0:root:/root:/bin/bash
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin

Cut out a section of a document

• awk -F 'delimiter' If no delimiter is specified, it will be separated by spaces or whitespace characters by default.

1. print the first paragraph

[root@linux-128 ~]# head -5 1.txt |awk -F ':' '{print $1}'
root
bin
daemon
adm
lp

1. Print the entire content; $0 means the entire content

[root@linux-128 ~]# head -5 1.txt |awk -F ':' '{print $0}'
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

1. Print paragraphs 1 to 4

[root@linux-128 ~]# head -5 1.txt |awk -F ':' '{print $1,$2,$3,$4}'
root x 0 0
bin x 1 1
daemon x 2 2
adm x 3 4
lp x 4 7 

• Note: The first to fourth paragraphs are printed here, but the content has no separator. Here we add a separator. The separator should be enclosed in double quotation marks; we will introduce a variable OFS to separate it later.

[root@linux-128 ~]# head -5 1.txt |awk -F ':' '{print $1"#"$2"#"$3"#"$4}'
root#x#0#0
bin#x#1#1
daemon#x#2#2
adm#x#3#4
lp#x#4#7

• awk can support multiple matches

1. Matching to root prints the 1st and 3rd paragraphs, and matching to the user prints the 1st, 3rd, and 4th paragraphs.

[root@linux-128 ~]# awk -F ':' '/root/{print $1,$3} /user/ {print $1,$3,$4}' 1.txt
root 0
operator 11
tss 59 59
user3 1004 1003
user4 1005 1003
user5 1007 1006
user6 1008 1003
user7 1009 1009

• |In the regular expression, or

1. print lines matching root or user

[root@linux-128 ~]# awk '/root|bash/' 1.txt
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
user7:x:1009:1009::/home/user7:/bin/bash

1. || also means and, but the usage is still a little different,

[root@linux-128 ~]# awk '/root/||/bash/' 1.txt
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
user7:x:1009:1009::/home/user7:/bin/bash

Conditional operator

• == equals, exact match

• more than the

• = greater than or equal to

• < less than
• <= less than or equal to
• != does not equal

1. print out the line where paragraph 3 equals 0

[root@linux-128 ~]# awk -F ':' '$3=='0'' 1.txt

1. Print out the lines where paragraph 3 is greater than or equal to 500

[root@linux-128 ~]# awk -F ':' '$3>='500'' 1.txt
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin
wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
user3:x:1004:1003::/home/user3:/sbin/nolongin
user4:x:1005:1003::/home/user4:/sbin/nolongin
user5:x:1007:1006::/home/user5:/bin/login
user6:x:1008:1003::/home/wuzhou:/sbin/nologin
user7:x:1009:1009::/home/user7:/bin/bash

1. Note the difference between '500' and "500": the 500 enclosed in double quotation marks will be treated as a character and will be sorted according to Asma, and the 500 enclosed by a single number is a number.

[root@linux-128 ~]# awk -F ':' '$3>="500"' 1.txt
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin

1. print out the line that does not /sbin/nologin in paragraph 7

[root@linux-128 ~]# awk -F ':' '$7!="/sbin/nologin"' 1.txt
root:x:0:0:root:/root:/bin/bash
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
user3:x:1004:1003::/home/user3:/sbin/nolongin
user4:x:1005:1003::/home/user4:/sbin/nolongin
user5:x:1007:1006::/home/user5:/bin/login
user7:x:1009:1009::/home/user7:/bin/bash

1. and && and or ||

[root@linux-128 ~]# awk -F ':' '$3>'1000' && $3<'1009'' 1.txt
user3:x:1004:1003::/home/user3:/sbin/nolongin
user4:x:1005:1003::/home/user4:/sbin/nolongin
user5:x:1007:1006::/home/user5:/bin/login
user6:x:1008:1003::/home/wuzhou:/sbin/nologin

1. or ||

[root@linux-128 ~]# awk -F ':' '$3>'1000' || $7=="/bin/bash"' 1.txt
root:x:0:0:root:/root:/bin/bash
wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
user3:x:1004:1003::/home/user3:/sbin/nolongin
user4:x:1005:1003::/home/user4:/sbin/nolongin
user5:x:1007:1006::/home/user5:/bin/login
user6:x:1008:1003::/home/wuzhou:/sbin/nologin
user7:x:1009:1009::/home/user7:/bin/bash

1. operation on two fields

[root@linux-128 ~]# awk -F ':' '$3>$4' 1.txt
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
operator:x:11:0:operator:/root:/sbin/nologin
systemd-bus-proxy:x:999:997:systemd Bus Proxy:/:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
chrony:x:997:995::/var/lib/chrony:/sbin/nologin
user3:x:1004:1003::/home/user3:/sbin/nolongin
user4:x:1005:1003::/home/user4:/sbin/nolongin
user5:x:1007:1006::/home/user5:/bin/login
user6:x:1008:1003::/home/wuzhou:/sbin/nologin

awk built-in variables

• Commonly used variables in awk are OFS, NR and NF
• OFS is used to specify the separator
• NR is the number of rows
• NF is the number of segments

1. Specify # as the delimiter

[root@linux-128 ~]# awk -F ':' '{OFS="#"} $3>"5" {print $1,$2,$3,$4}' 1.txt
shutdown#x#6#0
halt#x#7#0
mail#x#8#12
nobody#x#99#99
systemd-bus-proxy#x#999#997
dbus#x#81#81
polkitd#x#998#996
tss#x#59#59
postfix#x#89#89
sshd#x#74#74
chrony#x#997#995

1. Print first 5 lines with awk, and show line numbers

[root@linux-128 ~]# head -5 1.txt|awk -F ':' '{print NR":"$0}'
1:root:x:0:0:root:/root:/bin/bash
2:bin:x:1:1:bin:/bin:/sbin/nologin
3:daemon:x:2:2:daemon:/sbin:/sbin/nologin
4:adm:x:3:4:adm:/var/adm:/sbin/nologin
5:lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

1. Print lines after 20 lines with awk and display line numbers

[root@linux-128 ~]# awk -F ':' 'NR>20 {print NR":"$0}' 1.txt
21:chrony:x:997:995::/var/lib/chrony:/sbin/nologin
22:wuzhou:x:1000:1000::/home/wuzhou:/bin/bash
23:user3:x:1004:1003::/home/user3:/sbin/nolongin
24:user4:x:1005:1003::/home/user4:/sbin/nolongin
25:user5:x:1007:1006::/home/user5:/bin/login
26:user6:x:1008:1003::/home/wuzhou:/sbin/nologin
27:user7:x:1009:1009::/home/user7:/bin/bash

1. awk can use if to judge

[root@linux-128 ~]# awk -F ':' '{OFS="#"} {if($3>'1000'){print $1,$2,$3}}' 1.txt
user3#x#1004
user4#x#1005
user5#x#1007
user6#x#1008
user7#x#1009

Mathematical operations in awk

• awk can change segment value

1. Change the first paragraph all to root

[root@linux-128 ~]# head -5 1.txt|awk -F ':' '$1="root"'
root x 0 0 root /root /bin/bash
root x 1 1 bin /bin /sbin/nologin
root x 2 2 daemon /sbin /sbin/nologin
root x 3 4 adm /var/adm /sbin/nologin
root x 4 7 lp /var/spool/lpd /sbin/nologin

• awk can calculate the value of a segment

1. Find paragraph 3 and

[root@linux-128 ~]# awk -F ':' '{(oto=oto+$3)}; END {print oto}' 1.txt
9694

1. print out the line where the first paragraph is root

[root@linux-128 ~]# awk -F ':' '{if ($1=="root") {print $0}}' 1.txt
root:x:0:0:root:/root:/bin/bash