The usage examples of the sed tool and awk tool of shell programming are simple to understand with examples

One, understand the sed tool

1. Introduction to sed tool

sed (Stream EDitor) is a powerful and simple text parsing and conversion tool that can read the text and edit the text content (delete, replace, add, move, etc.) according to the specified conditions, and finally output all lines or only output processing Certain lines. Sed can also implement quite complex text processing operations without interaction, and is widely used in Shell scripts to complete various automated processing tasks.

Sed is a stream editor, the stream editor will edit the data stream based on a set of rules provided in advance before editing the processing data.

The sed editor can process the data in the data stream according to commands, which are either input from the command line or stored in a command text file. The work flow of
sed mainly includes three processes of reading, executing and displaying.

• Read : sed reads a line of content from the input stream (file, pipe, standard input) and stores it in a temporary buffer (also known as pattern space).
• Execution : By default, all sed commands are executed sequentially in the pattern space. Unless the address of the line is specified, the sed command will be executed sequentially on all lines.
• Display : Send the modified content to the output stream. After sending the data, the pattern space will be cleared.

2. Precautions for using sed

When using the sed command to process a file, the above process (read, execute, display) will be repeated until all the contents of the file are processed.

By default, all sed commands are executed in the cache space , so the file itself will not be changed due to the sed command , unless the redirection is used to output the edited file from the new storage to the new file .
For example, use the sed command to delete all the information in the /etc/passwd file. After rechecking the content in the etc/passwd file, it is found that it has not been deleted.

[root@localhost ~]# sed 'd' /etc/passwd      //使用sed命名删除/etc/passwd中所有的内容
[root@localhost ~]# cat /etc/passwd           //查看etc/passwd中的内容  
root:x:0:0:root:/root:/bin/bash          //所有内容都还在  说明使用sed命令不能改变源文件
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
...........        省略部分内容

3. Common usage of sed command

Usually there are two formats for calling the sed command:
sed [option]'operation' parameter
sed [option] -f scriptfile parameter

• Among them, "parameter" refers to the target file of the operation, which is used when there are multiple operation objects, and the files are separated by a comma ",";
• And scriptfile represents a script file, which needs to be specified with the "-f" option. When the script file appears before the target file, it means that the input target file is processed through the specified script file.

4. Common sed command options

Options	Comment
-f or --file=	Indicates that the specified script file is used to process the input text file.
-e or --expression=	Indicates that the specified command or script is used to process the input text file.
-h or --help	Show help.
--Quiet or silent	Indicates that only the processed results are displayed.
-i	Edit text files directly.

5. Common sed command operations

operating	Comment
a	Add, add a line of specified content below the current line.
c	Replace, replace the selected line with the specified content.
d	Delete, delete the selected row.
i	Insert, insert a line of specified content above the selected line.
p	Print, if you specify a line at the same time, it means to print the specified line; if you do not specify a line, it means to print all the content; if there are non-printable characters, it will be output in ASCII code. It is usually used with the "-n" option.
s	Replace, replace the specified character.
Y	Character conversion.
l	Print the text and non-printable ASCII characters in the data stream (such as the terminator %, the tab character \t)

Second, the usage example of sed tool

1. Output the specified line (p means normal output)

1.1, the basic usage of sed

[root@localhost ~]# sed -n 'p' /etc/passwd   //输出所有内容,等同于 cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
..........        //省略部分信息
[root@localhost ~]# sed -n '3p' /etc/passwd    //输出第三行的内容
daemon:x:2:2:daemon:/sbin:/sbin/nologin

[root@localhost ~]# sed -n '3,5p' /etc/passwd   //输出3到5行的内容
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

[root@localhost ~]# sed -n 'p;n' /etc/passwd 
//输出文本的奇数行，p打印第一行，n表示读入下一行的内容，没有输出，以此类推，就输出了全部的奇数行
root:x:0:0:root:/root:/bin/bash
daemon:x:2:2:daemon:/sbin:/sbin/nologin
..........        //省略部分信息

[root@localhost ~]# sed -n 'n;p'  /etc/passwd
//输出文本的偶数行，n表示读入下一行的内容，没有输出，p打印第二行的内容，以此类推，就输出了全部的偶数行
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
..........        //省略部分信息

[root@localhost ~]# sed -n '1,5{p;n}' /etc/passwd      //输出第 1~5 行之间的奇数行(第 1、3、5 行) 

[root@localhost ~]# sed -n '10,${n;p}' /etc/passwd    //输出第 10 行至文件尾之间的偶数行
//在执行“sed -n‘10,${n;p}’test.txt”命令时，读取的第 1 行是文件的第 10 行，
//读取的第 2行是文件的第 11 行，依此类推，所以输出的偶数行是文件的第 11 行、13 行直至文件结尾， 
//其中包括空行。

1.2, the usage of sed combined with regular expressions

[root@localhost ~]# sed -n '/root/p' /etc/passwd     //输出含有root字段的行
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

[root@localhost ~]# sed -n '2,/root/p' /etc/passwd  //输出从第2 行开始直到含有root的的行停止
bin:x:1:1:bin:/bin:/sbin/nologin  //第二行开始
..........        //省略部分信息
operator:x:11:0:operator:/root:/sbin/nologin   //直到含有root的行停止输出
        
[root@localhost ~]# sed -n '/root/=' /etc/passwd   //显示含有root行的行号
1
10

[root@localhost ~]# sed -n '/[0-5]/p' /etc/passwd     //输出含有数字1到5之间的行
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin 
..........        //省略部分信息     

[root@localhost ~]# sed -n '/\<root\>/p' /etc/passwd     //输出包含单词root的行, \<、\> （固定格式）代表单词边界
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin                                         

2. Delete eligible rows (d delete)

[root@localhost ~]# sed 'd' /etc/passwd   //删除所有行

[root@localhost ~]# sed '5d' /etc/passwd  //删除第5行

[root@localhost ~]# sed '5,10d' /etc/passwd  //删除5到10行

[root@localhost ~]# sed '/root/d' /etc/passwd  //删除含有root的字符的行

[root@localhost ~]# sed '/root/! d' /etc/passwd   //删除除了包含root的所有行

[root@localhost ~]# sed '/^[A-Z]/d' /etc/passwd   //删除以大写字母开头的行

[root@localhost ~]# sed '/[a-z]$/d' /etc/passwd  //删除以小写字母结尾的行

[root@localhost ~]# sed '/^$/d' /etc/passwd	//删除所有空行

[root@localhost ~]# sed '/root/,/the/d' /etc/passwd  
//从匹配第一个位置打开删除行功能，到匹配到第二个位置关闭删除功能
[root@localhost ~]# sed -e '/^$/{n;/^$/d}'  /etc/passwd 
//删除重复的空行，即连续的空行只保留一个，只能用于删除连续两行的空行中的一行。

3. Replace the qualified content (s (string replacement))

[root@localhost ~]# sed 's/^/#/' /etc/passwd	//在每行行首插入#号

[root@localhost ~]# sed 's/$/#/' /etc/passwd	//在每行行尾插入#

[root@localhost ~]# sed 's/root/ROOT/g' /etc/passwd	//将文件中的所有root 替换为 ROOT
//g:表明新字符串将会替换所有匹配的地方
[root@localhost ~]# sed 's/i/I/2' /etc/passwd	//将每行中的第 2 个 i替换为 I
//数字：表明新字符串将替换第几处的匹配的地方
[root@localhost ~]# sed 's/root/ROOT/' /etc/passwd	//将每行中的第一个root 替换为 ROOT

[root@localhost ~]# sed '5,10s/root/ROOT/g' /etc/passwd	//将第 5~10 行中的所有 root 替换为 ROOT

[root@localhost ~]# sed '/root/s/o/O/g' /etc/passwd	//将包含root 的所有行中的 o 都替换为 O

[root@localhost ~]# sed -n '9s/root/ROOT/p' /etc/passwd	//将第 9行中的所有 root 替换为 ROOT
p：打印与替换命令匹配的行，与-n一起使用

4. Move eligible content

Move the commonly used parameters that meet the conditions

Commonly used parameters	Comment
H	Copy to clipboard
G	Overwrite/append the data in the clipboard to the specified line
w	Save as file
r	Read the specified file
a	Additional specifications

sed '3aNew' /etc/passwd//在第 3 行后插入一个新行,内容为New

sed '/the/aNew' /etc/passwd	//在包含the 的每行后插入一个新行,内容为 New

sed '3aXIN1\nXIN2' /etc/passwd	//在第 3 行后插入多行内容,中间的\n 表示换行

sed '/the/{H;d};$G' /etc/passwd	//将包含the 的行迁移至文件末尾,{;}用于多个操作

sed '1,5{H;d};17G' /etc/passwd	//将第 1~5 行内容转移至第 17 行后

sed '/root/w wu.txt' /etc/passwd	//将包含the 的行另存为文件 wu.txt

sed '/the/r /etc/hostname' /etc/passwd	//将文件/etc/hostname 的内容添加到包含 the 的每行以后

5. Use scripts to edit files

[root@localhost mail]# sed '5,10{H;d};17G' /etc/passwd 	//将第 5~10 行内容转移至第 25 行后
[root@localhost mail]# vim 55    //编辑一个简单的操作动作文件
5,10H   //将5到10行复制到剪切板
5,10d   //将5到10行删除
25G     //将剪切板的内容粘贴到25行的后面
[root@localhost mail]#  sed -f 55 /etc/passwd    //通过sed调用文件完成编辑
..........        //省略部分信息 

6. Sed script application case

Write a script to adjust the vsftpd service configuration. It requires that anonymous users are prohibited, but local users are allowed to log in and have write permissions.

[root@localhost ~]# vim ftp.sh
#!/bin/bash
//配置文件路径
CONFIG="/etc/vsftpd/vsftpd.conf"   
//cp 命令对源行文件备份
[ -e "$CONFIG.old" ] || cp $CONFIG $CONFIG.old
//修改配置文件
sed -e '/^anonymous_enable/s/YES/NO/g' $CONFIG  //禁止匿名用户
sed -i -e '/^local_enable/s/NO/YES/g' -e '/^write_enable/s/NO/YES/g' $CONFIG//允许本地用户读写
sed -i '/^lisent/s/NO/YES/g' $CONFIG  //开启监听端口
//启动vsftpd 服务,并设为开机后自动运行
systemctl restart vsftpd 
systemctl enable vsftpd

Three, understand the awk tool

1. Awk commonly used format

Under normal circumstances, the command format used by awk is as follows, where single quotation marks and braces "{}" are used to set the processing actions on the data. Awk can process the target file directly, or read the script through "-f" to process the target file.

awk 选项 '模式或条件 {编辑指令}' 文件 1 文件 2 …  //过滤并输出文件中符合条件的内容
awk   -f   脚本文件 文件 1 文件 2 …	  //从脚本中调用编辑指令,过滤并输出内容

2. How awk is executed

As mentioned earlier, the sed command is often used to process a whole line, while awk tends to divide a line into multiple "fields" and then process it, and by default the field separator is a space or tab key. The awk execution result can be printed and displayed through the print function. In the process of using the awk command, you can use the logical operators "&&" to mean "and", "||" to mean "or", and "!" to mean "not"; you can also perform simple mathematical operations, such as +,- , *, /, %, ^ represent addition, subtraction, multiplication, division, remainder and power respectively.

Awk reads information from the input file or standard input. Like sed, the information is read line by line. The difference is that awk regards a line in the text file as a record, and regards a certain part (column) in a line as a field (domain) in the record. In order to manipulate these different fields, awk borrows a method similar to positional variables in the shell, using $1, $2, $3... to indicate the different fields in the row (record) sequentially. In addition, awk uses $0 to represent the entire line (record). Different fields are separated by specified characters. The default delimiter in awk is a space. Awk allows you to specify the separator in the form of "-F separator" on the command line.
For example, when executing the awk -F':''{print $1,$3,$4}' /etc/passwd command, the processing flow is as shown in the figure below. The most sufficient result is obtained by one behavior.

3. Common built-in variables that awk can use directly

Built-in variables	Comment
FS	Specify the field separator for each line of text, the default is a space or a tab stop.
NF	The number of fields in the currently processed row.
NO	The row number (ordinal number) of the row currently being processed.
$0	The entire line of the currently processed line.
$n	The nth field (nth column) of the currently processed row.
FILENAME	The name of the file being processed.
RS	Data records are separated, the default is \n, that is, one record per line.

Fourth, the usage example of awk tool

1. Output text by line

Output all content, equivalent to cat test.txt

[root@localhost ~]# awk '{print}' /etc/passwd     //输出全部行
[root@localhost ~]# awk '{print $0}' /etc/passwd    //输出全部行

Output a fixed number of lines

[root@localhost ~]# awk 'NR==1,NR==3{print}' /etc/passwd         //输出第 1~3 行内容
[root@localhost ~]# awk '(NR>=1)&&(NR<=3){print}' /etc/passwd  	//输出第 1~3 行内容
[root@localhost ~]# awk 'NR==1||NR==3{print}' /etc/passwd    	//输出第 1 行、第 3 行内容

Output base and even lines

[root@localhost ~]# awk '(NR%2)==1{print}' test.txt	//输出所有奇数行的内容
[root@localhost ~]# awk '(NR%2)==0{print}' test.txt	//输出所有偶数行的内容

What line starts and ends with output

[root@localhost ~]# awk '/^root/{print}' /etc/passwd	//输出以root 开头的行
[root@localhost ~]# awk '/nologin$/{print}' /etc/passwd	//输出以 nologin 结尾的行

Count the number of rows filtered by awk

[root@localhost ~]# awk 'BEGIN {x=0};/\/bin\/bash$/{x++};END {print x}' /etc/passwd
//awk执行的顺序是，首先执行BEGIN｛｝中的操作；然后从指定的文件中逐行读取数据，
自动更新NF、NR、$0、$1等内建变量值，如执行‘模式或条件｛编辑指令｝’；最后执行
END｛｝中的后续操作。
相当于开始位和结束位
//统计以/bin/bash 结尾的行数,等同于 grep -c "/bin/bash$" /etc/passwd
[root@localhost ~]# awk 'BEGIN{RS=""};END{print NR}' /etc/passwd
//统计以空行分隔的文本段落数

2. Output text by field

awk '{print $3}'  /etc/passwd          //输出每行中(以空格或制表位分隔)的第 3 个字段
awk '{print $1,$3}'  /etc/passwd	                //输出每行中的第 1、3 个字段
awk -F ":" '$2==""{print}' /etc/shadow             //输出密码为空的用户的shadow 记录
awk 'BEGIN {FS=":"}; $2==""{print}' /etc/shadow   //输出密码为空的用户的shadow 记录
awk -F ":" '{print NR, $0}' /etc/passwd    //输出处理数据的行号，每处理完一条记录，NR值加一
awk -F ":" '$3<5 {print $1 $3}' /etc/passwd  //输出第三列小于5的第一列与第三列的数据
awk -F ":" 'NR==1,NR==7{print $1,$7}' /etc/passwd  
//输出第三行到第七行中以冒号为分割符的第1列和第7列的数据
awk -F ":" '$7~"/bash"{print $1}' /etc/passwd
//输出以冒号分隔且第 7 个字段中包含/bash 的行的第 1 个字段
awk '($1~"nfs")&&(NF==7){print $1,$2}' /etc/passwd
//输出包含 7 个字段且第 1 个字段中包含 nfs 的行的第 1、2 个字段
awk -F ":" '($7!="/bin/bash")&&($7!="/sbin/nologin"){print}' /etc/passwd
//输出第 7 个字段既不为/bin/bash 也不为/sbin/nologin 的所有行

  输出的数据可以插入文本标签，用双引号括起来放到变量前
awk -F ":" 'NR==1,NR==7{print "第1位:"$1,"第7位:"$7}' /etc/passwd                  
第1位:root 第7位:/bin/bash  
..........        //省略部分信息   

3. Awk uses conditional expressions

awk -F ":" '{if($3>200){print $-}}' /etc/passwd  //输出第3个字段大于200的行
awk -F ":" '{max=($3>$4) ? $3:$4; print max}' /etc/passwd
//如果第3个字段的值大于第4个字段的值，则把问好前表达式的值赋给max，否者就就将
冒号后那个表达式的值赋给max
awk -F ":" '{max=($3>200)? $3:$1;print max}' /etc/passwd
//如果第3个字段的值大于200,则把第3个字段的值赋给max，否者就将第1个字段的值赋给max

4. Awk uses the pipe symbol

awk -F: '/bash$/{print | "wc -l"}' /etc/passwd
//调用wc -l 命令统计使用 bash 的用户个数,等同于 grep -c "bash$" /etc/passwd
awk 'BEGIN {while ("w" | getline) n++ ; {print n-2}}'
//调用w 命令,并用来统计在线用户数
awk 'BEGIN { "hostname" | getline ; print $0}'
//调用hostname,并输出当前的主机名
//当getline左右无重定向符"<"或"|"时，getline则作用于当前文件，读入当前文件的第一行给其后跟的变量var或$1；
应该注意到，由于awk在处理getline之前已经读入了一行，所以getline得到的返回结果是隔行的。
当getline左右有重定向符"<"或"|"时，getline则作用于定义输入文件，由于该文件是刚打开的，
并没有被awk读入一行，只是getline读入，那么getline返回的是该文件的第一行，而不是隔行。

5. Awk custom separator filter information

For example; the string "http://www.55xuexi.ed/root/456.html" in the text 55.txt; filter out "http:", "www.55xuexi.eda", "root", "456.htm" and "456" are output.

[root@localhost ~]# awk -F "//" '{print $1}' /root/55.txt
http:
[root@localhost ~]# awk -F "/" '{print $3}' /root/55.txt  
www.55xuexi.ed
[root@localhost ~]# awk -F "/" '{print $4}' /root/55.txt 
root
[root@localhost ~]# awk -F "root/" '{print $2}' /root/55.txt 
456.html
[root@localhost ~]# awk -F "." '{print $3}' /root/55.txt|awk -F "/" '{print $3}'
456

6. Awk script application case

The script is used to send an E-mail alarm when the use of disk space exceeds 90%.

#!/bin/bash
#监视可用磁盘空间
#monitor available disk space
#截取以“/”为结尾的行，打印第5个字段也就是跟分区使用百分比，截取掉“%”
K=`df | sed -ne '/\/$/ p' | awk '{print $5}'| sed 's/%//'`
if [ $K -ge 90 ]    //如果占比超过90就向管理员发报警邮件
then
echo "磁盘可用空间报警" | mail -s "磁盘报警"[email protected]
fi
//可以将此脚本放到周期性任务列表里定时检查磁盘空间使用情况