Shell programming extended regular expressions (egrep, awk)

Extended regular expression

Extended regular expression metacharacters

• Extended regular expressions are the expansion and deepening of basic regular expressions

Extended metacharacters

• +: Match the previous sub-expression more than once, for example: go+d, will match at least one o (such as god, good, goood...)
• ?: Match the previous sub-expression 0 or 1 time For example: go?d, will match gd or god
• (): The string in the brackets as a whole, for example: (xyz)+, will match xyz as a whole more than once, such as xyzxyz
• |: Match the string of words in an or manner. For example, 1: good|food, will match good or food, for example 2: g(oo|la)d, will match good or glad
  (g|f)ood good or food

egrep

Example

Match o content that appears at least once or more times

[root@server2 ~]# vi test.txt 
[root@server2 ~]# grep -n 'oo*' test.txt 

[root@server2 ~]# vi test.txt 
[root@server2 ~]# egrep -n 'o+' test.txt 

The match starts with be, ends with t, and can be s or empty content in the middle

[root@server2 ~]# egrep -n 'bes?t' test.txt 

The match starts with sh, ends with rt, and the content can be i or o in the middle

[root@server2 ~]# grep -n sh[io]rt test.txt 
[root@server2 ~]# egrep -n 'sh(i|o)rt' test.txt 
[root@server2 ~]# egrep -n sh'(i|o)'rt test.txt 

Matching contains content that oo appears once or more than once

[root@server2 ~]# egrep -n '(oo)+' test.txt 

Introduction to awk tools

format

• awk option'mode or condition {edit instruction}' file 1 file 2
• awk -f script file file 1 file 2

Parsing

Awk is a powerful editing tool. It reads the input text line by line and searches according to the specified matching mode. It formats and filters the content that meets the conditions. It can realize quite complex text without interaction. Operation is widely used in Shell scripts to complete various automated configuration tasks.

How awk works

Read text line by line, separated by space by default, save the separated fields to built-in variables, and execute editing commands according to the mode or condition

Built-in variables

• FS: Specify the field separator for each line of text, the default is a space or tab stop.
• NF: The number of fields in the currently processed row.
• NR: The row number (ordinal number) of the row currently being processed.
• $0: The entire line content of the currently processed line.
• $n: The nth field (nth column) of the currently processed row.
• FILENAME: The name of the file being processed
• RS: Data record separation, the default is \n, that is, one record per row

usage

Find out the first and seventh field contents of all file information in /etc/passwd

[root@server2 ~]# awk -F: '{print $1,$7}' /etc/passwd

Find out the first and seventh field contents of all file information in /etc/passwd and start with zjq and end with /sbin/nologin

[root@server2 ~]# awk -F: '/^zjq/{print $1,$7}' /etc/passwd
[root@server2 ~]# awk -F: '/\/sbin\/nologin$/{print $1,$7}' 

Example of built-in variable usage

Output the first and seventh fields of the third line

[root@server2 ~]# awk -F: 'NR==3{print $1,$7}' /etc/passwd

Output the first and seventh fields from line 3 to line 6, output the third line, the first and seventh fields of line 6, and output the first and seventh fields greater than or equal to line 3 and less than or equal to line 6

[root@server2 ~]# awk -F: 'NR==3,NR==6{print $1,$7}' /etc/passwd
[root@server2 ~]# awk -F: '(NR==3)||(NR==6){print $1,$7}' /etc/passwd
[root@server2 ~]# awk -F: '(NR>=3)&&(NR<=6){print $1,$7}' /etc/passwd

Output odd-numbered lines, even-numbered lines

[root@server2 ~]# cat /etc/passwd | wc -l 
[root@server2 ~]# awk -F: '(NR%2)==1{print $1,$7}' /etc/passwd | wc -l
[root@server2 ~]# awk -F: '(NR%2)==0{print $1,$7}' /etc/passwd | wc -l

Count the number of lines ending in /bin/bash

[root@server2 ~]# awk  'BEGIN {x=0};/\/bin\/bash$/{x++};END {print x}' /etc/passwd

The third field (separated by spaces or tabs) in each line of the output file

[root@server2 ~]# awk '{print $3}' test.txt 

The first and third fields (separated by spaces or tabs) in each line of the output file

[root@server2 ~]# awk '{print $1,$3}' test.txt 

Output users without a password

[root@server2 ~]# awk -F: '$2=="!!"{print}' /etc/shadow 
[root@server2 ~]# awk  'BEGIN {FS=":"};$2=="!!"{print}' /etc/shadow 

Output the users whose seventh field contains /bash, /bin/bash

[root@server2 ~]# awk -F: '$7~"/bash"{print $1}' /etc/passwd
[root@server2 ~]# awk -F: '$7~"/bin/bash"{print $1}' /etc/passwd

The first field of the output in /etc/services is nfs and the number of fields is 4

[root@server2 ~]# grep "nfs" /etc/services
[root@server2 ~]# awk '($1~"nfs")&&(NF==4){print $1}' /etc/services

The output of the seventh field is not /bin/bash nor /sbin/nologin

[root@server2 ~]# awk -F: '($7!="/bin/bash")&&($7!="/sbin/nologin"){print}' /etc/passwd

Call the wc -l command and use it to count 1 the number of lines ending with /bash, that is, the number of users

[root@server2 ~]# awk -F: '/bash$/{print}' /etc/passwd | wc -l
[root@server2 ~]# awk -F: '/bash$/{print | "wc -l"}' /etc/passwd 
[root@server2 ~]# grep -c "bash$" /etc/passwd

Call w command to count the number of online users

[root@server2 ~]# awk 'BEGIN {while ("w" | getline) n++; {print n-2}}'

Call hostname and output the current hostname

[root@server2 ~]# awk 'BEGIN {"hostname" | getline;print $0}' 

Output the value in the path variable, the value of the third field

[root@server2 ~]# echo $PATH
[root@server2 ~]# echo "$PATH" > a.txt
[root@server2 ~]# cat a.txt 
[root@server2 ~]# awk -F: '{print}' a.txt
[root@server2 ~]# awk -F: '{print$3}' a.txt