shell scripting of regular expressions (c)
I. Introduction
The previous two articles introduces the concept and role of regular expressions, basic regular expressions and extended regular expression to the concept of narrative; also introduced shell "Three Musketeers" of grep and sed two commands, this article will introduce Finally, a swordsman --awk.
We will also introduce tools and uniq sort tool.
Two, awk tool
In Linux / Unix systems, awk is a powerful tool for editing, reading row by row input text, and to find the matching based on the specified pattern to meet the requirements of the output format or content filtering process can not interact the case of implementing fairly complex text manipulation, are widely used in Shell script to complete a variety of automated configuration tasks.
Common usage
awk option 'edit mode or conditional instruction {}' file1 file2
awk -f script file 1 file 2
sed command used to process an entire row, awk sucked into multiple fields after the processing line, and the default field separator is a case where the space or the TAB key.
awk execution result can be displayed by the print data printing field function. In using awk command can use the logical operator "&&" indicates "and", "||" indicates "or" means "not", "!"; Also can perform simple mathematical operations, such as +, -, *, /,%, ^ respectively addition, subtraction, multiplication, division, exponentiation, and take the remainder.
awk contains several special built-in variables (can not be modified but can be used directly)
FS: Field Separator Specify each line of text, space or tab default bit (TAB)
The number of fields of the current processing of the row: NF
NR: line number of the currently processed (ordinal)
The contents of the current line of treatment: $ 0
$ N: n-th field of the current process line (n-th column)
FILENAME: File name being processed
RS: partition data record, default \ n i.e. each record behavior
Examples about:
1) Find a user name / etc / passwd file, user ID, group ID column
[root@lokott ~]# awk -F: '{print $1,$3,$4}' /etc/passwd
root 0 0
bin 1 1
daemon 2 2
...//省略部分内容
[root@lokott ~]# awk -F: '{print $0}' /etc/passwd //$0表示显示整个行，结果与cat 和sed -n 'p'等价
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
......//省略部分内容2) Press the OK output text
[root@lokott ~]# awk '{print}' /etc/passwd //等同于 awk '{print $0}' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
[root@lokott ~]# awk 'NR==1,NR==3{print}' /etc/passwd //输出1-3的行内容，与下等同
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@lokott ~]# awk '(NR>=1)&&(NR<=3){print}' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@lokott ~]# awk '(NR==1)||(NR==3){print}' /etc/passwd //输出第1和3行的内容
root:x:0:0:root:/root:/bin/bash
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@lokott ~]# nl /etc/passwd |awk 'NR%2==1{print}' //输出奇数行内容
1 root:x:0:0:root:/root:/bin/bash
3 daemon:x:2:2:daemon:/sbin:/sbin/nologin
5 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
7 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
......//省略部分内容
[root@lokott ~]# nl /etc/passwd |awk 'NR%2==0{print}' //输出偶数行内容
2 bin:x:1:1:bin:/bin:/sbin/nologin
4 adm:x:3:4:adm:/var/adm:/sbin/nologin
6 sync:x:5:0:sync:/sbin:/bin/sync
8 halt:x:7:0:halt:/sbin:/sbin/halt
[root@lokott ~]# awk '/^root/{print}' /etc/passwd //输出以root开头的行
root:x:0:0:root:/root:/bin/bash
[root@lokott ~]# awk '/bash$/{print}' /etc/passwd //输出以bash结尾的行
root:x:0:0:root:/root:/bin/bash
lokott:x:1000:1000:lokott:/home/lokott:/bin/bash
lisi:x:1002:1002::/home/lisi:/bin/bash
[root@lokott ~]# awk 'BEGIN{x=0};/\/bin\/bash$/ {x++};END {print x}' /etc/passwd
3
[root@lokott ~]# grep -c "/bin/bash$" /etc/passwd
3
//统计以/bin/bash结尾的行的数量3) Press the output text field
[root@lokott ~]# sed 's/:/ /g' /etc/passwd|awk '{print $1,$3}'|tail -3
named 25
apache 48
lisi 1002
//将passwd文件中的所有：替换为空格后的内容到缓存中，然后使用awk命令输出每行的第一和第三个字段到管道中，最后输出最后三行到显示屏上，也可以在后面继续添加“|sort”继续排序
[root@lokott ~]# awk -F: '$2=="!!"{print}' /etc/shadow |tail -3
tcpdump:!!:18214::::::
named:!!:18220::::::
apache:!!:18228::::::
//输出密码为！！（没有密码的）的用户的shadow记录
[root@lokott ~]# awk -F: '$7~"/bash"{print $1}' /etc/passwd//输出第七个字段中包含/bash的行的第一个字段
root
lokott
lisi
[root@lokott ~]# awk '($1~"nfs")&&(NF==8){print $1,$2}' /etc/services
nfs 2049/tcp //输出包含8个字段且第一个字段包含nfs的行的第1、2个字段
nfs 2049/udp
nfs 2049/sctp
netconfsoaphttp 832/tcp
netconfsoaphttp 832/udp
netconfsoapbeep 833/tcp
netconfsoapbeep 833/udp
4) through the pipe, shell commands invoke double quotes
[root@lokott ~]# awk '/bash$/{print |"wc -l"}' /etc/passwd
3
//输出使用bash的用户个数Three, sort and uniq tools Tools
On Linux systems, commonly used file sorting tool, there are three: sort, uniq, wc. wc command chapters explain in linux, and for general statistics, and therefore will not repeat them here. Sort and uniq mainly on the use of tools and methods of use.
1.sort
sort is a tool in units of the contents of the file to be sorted, it can also be sorted according to different data types.
Using the format: sort [option] parameter
Options:
- -f: ignore case;
- -b: Ignore spaces in front of each row;
- -M: Sort by month;
- -n: sort numerically;
- -r: Reverse order; (common)
- -u: equivalent to uniq, showing the same data as the single line;
- -t: Specifies delimiter, using [Tab] key default partition;
- -o <output file>: The sorted results to the designated dump file; (common)
- -k: Specifies the sorting area.
Example:
[root@lokott ~]# sort /etc/passwd |nl //依据字母排序正向
1 abrt:x:173:173::/etc/abrt:/sbin/nologin
2 adm:x:3:4:adm:/var/adm:/sbin/nologin
3 apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
4 avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
5 bin:x:1:1:bin:/bin:/sbin/nologin
......//省略部分内容
[root@lokott ~]# sort -r /etc/passwd |nl //依据字母排序反向
1 usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
2 tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
3 tcpdump:x:72:72::/:/sbin/nologin
4 systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
5 sync:x:5:0:sync:/sbin:/bin/sync
......//省略部分内容
[root@lokott ~]# sort -t: -k 1 /etc/passwd //根据：号将第一列进行正向排序
abrt:x:173:173::/etc/abrt:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
chrony:x:995:991::/var/lib/chrony:/sbin/nologin
colord:x:997:995:User for colord:/var/lib/colord:/sbin/nologin
......//省略部分内容
[root@lokott ~]# sort -t: -k 1 /etc/passwd -o out.txt //将上述结果输入到文件out.txt中
[root@lokott ~]# cat out.txt
abrt:x:173:173::/etc/abrt:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
......//省略部分内容2.uniq
uniq tool often used in conjunction with the sort command in Linux systems, for reporting or ignore file duplicates.
Format: uniq [options] parameters
Options:
-c: count the
-d: show only duplicate lines
-u: display only appears once in row
Example:
[root@lokott ~]# cat test.txt //有空行
this is a test
this is a test
this is a test
hello
hello
world
this is a test
this is a test
list
list
hostname
hostname
[root@lokott ~]# uniq test.txt //删除互相连续的重复的行
this is a test
hello
world
this is a test
list
hostname
[root@lokott ~]#uniq -c test.txt //行前显示该行重复出现的此次数
3 this is a test
1
2 hello
1
1 world
2
2 this is a test
2 list
1
2 hostname
1
[root@lokott ~]# uniq -d test.txt //删除连续重复的重复行
this is a test
hello
this is a test
list
hostname
IV Summary
This article explains that the use and methods of use awk, sort and uniq tool, explained with examples.