Shell Programming Three Musketeers --awk regular expression tools

awk Overview

In Linux / UNIX systems, awk is a powerful tool for editing, the input text is read line by line, and to find the matching according to the specified pattern to meet the requirements of the output format or content filtering process can not interact the case of implementing fairly complex text manipulation, are widely used in Shell script to complete a variety of automated configuration tasks.

1, awk common usage

Typically used awk command format is shown below in the case in which, coupled with single quotes braces "{}" is provided for processing the data of the operation. awk can deal directly with the target file, the target file can be processed by the "-f" read the script.

awk option 'edit mode or conditional instruction {}' file1 file2 "// file and outputs the filtered content identifier conditions

awk -f script file 1 file 2 "// Call command from the script editor, content filtering and output

Sed command used in the aforementioned process on one row, the row will tend awk into a plurality of "field" and then processed, and the default field separator is a space or tab keys. awk execution result can be displayed by the print data printing field function. In using awk command can use the logical operator "&&" indicates "and", "||" indicates "or" means "not", "!"; Also can perform simple mathematical operations, such as +, -, *, /,%, ^ respectively addition, subtraction, multiplication, division, exponentiation, and take the remainder.

In the Linux system / etc / passwd is a very typical file format, each field using the ":" as the delimiter separated, most of the log file is formatted Linux file system, extract the relevant information from these files It is part of the daily operation and maintenance of the work content. If you need to find out the username / etc / passwd, user ID, group ID, column, execute the following command to awk.

awk -F ':' '{print $1,$3,$4}' /etc/passwd

[root@localhost ~]# awk -F : '{print $1,$2,$3}' /etc/passwd
root x 0
bin x 1
daemon x 2
adm x 3

awk reads from the standard input file or input information, and as sed, the information is read line by line read. Awk except that the line of the text file as a record, and a part of the row (column) as a field (field) in a record. In order to operate these various fields, a method shell similar to the position of the variable awk borrowed by $ 1, $ 2, $ 3 "sequentially showing lines of different fields (records). Further awk by $ 0 represents the entire row (record). Different fields are separated by character specified .awk default delimiter is a space in the form of command line allows .awk by "-F separator" to specify the delimiter. Thus, the above-described example, awk command to / etc / FIG passwd file processing

awk contains several special built-in variables (which can be directly used) as follows: FS: Specify each line of text field separator, the default position is a space or tab.

NF: number of columns of the row currently being processed.
NR: line number of the currently processed (ordinal).
$ 0: The entire line row currently being processed.
$ n: n-th field of the current process line (column n).
FILENAME: File name to be processed.
RS: recording data separated by default \ n-, i.e. the behavior of each record.

2, usage examples

1) row output text

awk '{print}' test.txt // output all content, equivalent to test.txt CAT
awk '{0} Print $' test.txt // output all content, equivalent to test.txt CAT
awk '== NR. 1, NR == 3 {print} 'test.txt // output lines 1 to 3 content
awk' (NR> = 1) && (NR <= 3) {print} 'test.txt // output lines 1 to 3 SUMMARY
awk 'NR == 1 || NR == 3 {print}' test.txt // output line 1, line 3 content
awk '(NR% 2) == 1 {print}' test.txt // the outputs of all odd row content
awk '(NR% 2) == 0 {print}' test.txt // output the contents of all even rows
awk '/ ^ root / {print }' / etc / passwd // root output beginning of the line
awk '/ nologin $ / {print }' / etc / passwd // output nologin end line
awk 'BEGIN {x = 0} ; / \ / bin \ / bash $ / {x ++}; eND {print } X '/ etc / the passwd
// statistics to / bin rows / bash end is equivalent to -C grep "/ bin / bash $" / etc / the passwd
awk' the BEGIN {the RS = ""}; {Print the eND NR } '/etc/squid/squid.conf
// count the number of paragraphs with blank lines delimited text

2) Press the output text field

awk '{print $ 3}' test.txt // output each line (separated by a space or tab stop) of three fields
awk '{print $ 1, $ 3}' test.txt // outputs per row 1,3 fields

[root@localhost ~]# awk '{print $3}' test.txt.bak 

best

cross

[root@localhost ~]# awk '{print $1,$3}' test.txt.bak 
the 
you best
PI=3.1415926535897

awk -F ":" '$ 2 == "" {print}' / etc / shadow // output a blank password shadow record user
awk 'BEGIN {FS = ": "}; $ 2 == "" {print} '/ etc / shadow // output a blank password shadow user records

[root@localhost ~]# awk -F ":" '$2==""{print}' /etc/shadow
test::18179:0:99999:7:::
[root@localhost ~]# awk 'BEGIN {FS=":"}; $2==""{print}' /etc/shadow
test::18179:0:99999:7:::

awk -F ":" '$. 7 ~ "/ bash" {Print $. 1}' / etc / the passwd
// output separated by a colon and a seventh field comprises a first field line / bash of
awk '($ 1 ~ " nfs ") && (of NF ==. 8). 1 {Print $, $ 2} '/ etc / Services
// contains eight fields and the output of the first and second fields a field comprising rows of nfs
awk -F": " '($. 7! =" / bin / bash ") && ($. 7! =" / sbin / nologin ") {Print}' / etc / the passwd
// output field 7 is neither / bin / bash is not / sbin / nologin all rows

[root@localhost ~]# awk -F ":" '$7~"/bash"{print $1}' /etc/passwd
root
test1
lisi
tom
test
[root@localhost ~]# awk '($1~"nfs")&&(NF==8){print $1,$2}' /etc/services
nfs 2049/tcp
nfs 2049/udp
nfs 2049/sctp
[root@localhost ~]# awk -F ":" '($7!="/bin/bash")&&($7!="/sbin/nologin"){print}' /etc/passwd
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt

3) through a pipe, invoke shell commands double quotes

awk -F: | '/ bash $ / {print "wc -l"}' / etc / passwd
number of user command wc -l // Call statistics using bash, equivalent to grep -c "bash $" / etc / the passwd
awk 'the BEGIN {the while ( "w" | getline) n-++; {Print n--2}}'
// call w command, and used to count the number of online users
awk 'the BEGIN { "hostname" | getline; Print $ 0}'
/ / call hostname, and the output current host name

[root@localhost ~]# awk -F: '/bash$/{print | "wc -l"}' /etc/passwd
5
[root@localhost ~]# awk 'BEGIN {while ("w" | getline) n++ ; {print n-2}}'
1
[root@localhost ~]# awk 'BEGIN { "hostname" | getline ; print $0}'
localhost.localdomain

thanks for reading! ! !