Contains many text processor or text editor in Linux / UNIX systems, which grep, sed and awk text processing tools are often used in shell programming, therefore, the industry of a wide range of people called "shell programming Three Musketeers " .
grep command tool
grep command tools in their daily lives, will be frequently used, there is not much to say, if you have friends do not understand, please refer Bowen: regular expressions Shell scripts use the Detailed , which detailed the grep command related parameters and use,
sed command tool
sed is a powerful and simple text analysis and conversion tool that can read the text, and edit text according to the specified conditions, the final output all lines live output processing only certain lines, sed can be achieved in the absence of interaction situations rather complex text processing operations . It is widely used in shell scripts for automated processing to complete various tasks.
sed的工作流程主要包括： 1. 读取：sed从输入流中读取一行内容不能够存储到临时的缓冲区中； 2. 执行：默认情况下所有的sed命令都在模式空间中按顺序地执行，除非指定了行的地址，否则sed命令将会再所有行上依次执行； 3. 显示：发送修改后的内容到输出流，再发送数据后，模式空间将会被清空。 注意：在所有的文件内容都被处理完成之前，上述过程将重复执行，直至所有内容都被处理完。
1) sed command syntax and parameters:
sed [选项] '操作' 参数 或 sed [选项] -f scriptfile 参数
The main parameters common sed command options: 
If you require to be modified between the first few lines to a few lines, etc., common operating parameters include:
2) sed command usage examples (note that the following actions will not change the content of the file itself, if you need to modify must take "-i" option)
1. Output matching texts
[root@localhost ~]# sed -n 'p' test.txt //输出所有内容，等同于“cat test.txt”
[root@localhost ~]# sed -n '3p' test.txt //输出第三行内容
[root@localhost ~]# sed -n '3,5p' test.txt //输出3~5行
[root@localhost ~]# sed -n 'p;n' test.txt //输出所有奇数行，n表示读入下一行数据
[root@localhost ~]# sed -n 'n;p' test.txt //输出所有偶数行，n表示读入下一行数据
[root@localhost ~]# sed -n '1,5{p;n}' test.txt //输出第1行~第5行之间的奇数行（第1、3、5行）
[root@localhost ~]# sed -n '10,${n;p}' test.txt //输出第10行至文件尾部之间的偶数行（包括空行）
sed command and regular expressions with case used
sed command in conjunction with a regular expression, the format is slightly different, the regular expression to "/" surrounded .
[root@localhost ~]# sed -n '/the/p' test.txt //输出包含“the”的行
[root@localhost ~]# sed -n '4,/the/p' test.txt //输出从第4行到都第一个包含“the”的行
[root@localhost ~]# sed -n '/the/=' test.txt //输出包含“the”的行所在的行号（等号（=）用来输出行号）
[root@localhost ~]# sed -n '/^PI/p' test.txt //输出以“PI”开头的行
[root@localhost ~]# sed -n '/\<wood\>/p' test.txt //输出包含单词wood的行，\<、\>代表单词边界
2. Delete the matching texts
Nl command is used to calculate the number of lines in the file
[root@localhost ~]# nl test.txt | sed '3d' //删除第3行
[root@localhost ~]# nl test.txt | sed '3,5d' //删除第3~5行
[root@localhost ~]# nl test.txt | sed '/cross/d' //删除包含cross的行，原本的第8行被删除
[root@localhost ~]# nl test.txt | sed '/cross/! d' //删除不包含cross的行
[root@localhost ~]# sed '/\.$/d' test.txt //删除以“.”结束的行 [root@localhost ~]# sed '/^$/d' test.txt //删除所有空行 [root@localhost ~]# sed -e '/^$/{n;/^$/d}' test.txt //删除空行，连续的空行留一个
3. Replace the qualifying text
Option will need to be replaced using sed command operation: S (string replacement), c (entire row / replacement block), y (character conversion ) command option and the like. Because the test file does not meet the requirements, the following is not a screenshot.
[root@localhost ~]# sed 's/the/THE/' test.txt //将每行中的第一个the替换为THE [root@localhost ~]# sed 's/l/L/2' test.txt //将每行中的第三个“l”替换为“L” [root@localhost ~]# sed 's/the/THE/g' test.txt //将文件中所有的“the”替换为“THE” [root@localhost ~]# sed 's/o//g' test.txt //将文件中所有的“o”删除 [root@localhost ~]# sed 's/^/#/' test.txt //在每行的行首插入“#”号 [root@localhost ~]# sed '/the/s/^/#/' test.txt //在包含“the”的每行行首插入“#”号 [root@localhost ~]# sed 's/$/EOF/' test.txt //在每行行尾插入字符串“EOF” [root@localhost ~]# sed '3,5s/the/THE/g' test.txt //将第3~5行中的所有“the”替换为“THE” [root@localhost ~]# sed '/the/s/o/O/g' test.txt //将包含“the”的所有行中的o替换为“O”
More "sed -i" command is effective immediately!
[root@localhost ~]# sed -i '1c 1111' a.txt //替换文中第一行的内容为“1111” [root@localhost ~]# sed -i '1a 1111' a.txt //在第一行后面插入一行内容，内容为“1111” [root@localhost ~]# sed -i '1i 2222' a.txt //在第一行前面插入一行内容，内容为“2222” [root@localhost ~]# sed -i '1d' a.txt //删除第一行内容 [root@localhost ~]# sed -n '1p' a.txt //打印出第一行的内容 [root@localhost ~]# sed -i '1s/2222/3333/g' a.txt //将文本第一行内容“2222”替换为“3333”
4. Migration qualifying text
Use sed command needed when migrating text manipulation options are: g, G the data covering the clipboard / appended to the specified line; w saved as a file; R & lt reads the specified file; A specifies additional content.
[root@localhost ~]# sed '/the/{H;d};$G' test.txt //将包含“the”的行迁移到文件末尾，“；”用于多个操作 [root@localhost ~]# sed '1,5{H;d};17G' test.txt //将第1~5行的内容转移到第17行后 [root@localhost ~]# sed '/the/w out.file' test.txt //将包含“the”的行另存为文件out.file [root@localhost ~]# sed '/the/r /etc/hostname' test.txt //将文件/etc/hostname的内容添加到包含“the”的每行以后 [root@localhost ~]# sed '3aNEW' test.txt //在第3行后面插入一个新行，内容为“NEW” [root@localhost ~]# sed '/the/aNEW' test.txt //在包含“the”的每行后插入一个新行，内容为“NEW” [root@localhost ~]# sed '3aNEW1\nNEW2' test.txt //在第3行后面多行内容，中间的“\n”表示换行
5. Edit the file using a script
Use sed script, the editing instruction stored in the file (one per line marking instructions), by "-f" option to call.
[root@localhost ~]# sed '1,5{H;d};17G' test.txt //将第1~5行内容转移至第17行后
The above operation is converted to a script file method:
[root@localhost ~]# vim 1.list 1,5H 1,5d 17G [root@localhost ~]# sed -f 1.list test.txt
Direct file exemplary operation 6.sed
Write a script, used to adjust the vsftpd service configuration: Disable anonymous user, but allows local users (also allow writing) login.
[root@localhost ~]# vim local_only_ftp.sh #!/bin/bash S="/usr/share/doc/vsftpd-3.0.2/EXAMPLE/INSERNET_SITE/vsftpd.conf" C="/etc/vsftpd/vsftpd.conf" #指定样本文件路径、配置文件路径 [ ! -e "$C.bak" ] && cp $C $C.bak #备份原来的配置文件，检测(配置文件.bak)是否存在，如果不存在则使用cp命令复制 sed -e '/^anonymous_enable/s/YES/NO/g' $S > $C sed -i -e '/^local_enable/s/NO/YES/g' -e '/^write_enable/s/NO/YES/g' $C grep "listen" $C || sed -i '$alisten=YES' $A #基于样本配置进行调整，覆盖现有文件 systemctl restart vsftpd systemctl enable vsftpd #重启ftp服务，并设置为开机自启动
awk command tool
In Linux / UNIX systems, awk is a powerful tool for editing, the input text is read line by line, to find the matching according to the specified pattern to meet the requirements of the output format or content filtering processing, can no the case of interaction to achieve fairly complex text manipulation , are widely used in Shell script to complete a variety of automated configuration tasks.
The syntax of the command and overview 1.awk
awk 选项 '模式或条件 { 编辑指令 }' 文件1 文件2 … //过滤并输出文件符合条件的内容 awk -f 脚本文件 文件1 文件2 … //从脚本中调用编辑指令，过滤并输出内容
* Awk execution results may be displayed by the print data printing field function. In using awk command you can use the logical operator "&&" and "||";
can be simple mathematical operations, such as +, - , /,%, ^ respectively addition, subtraction, multiplication, and division , modulo, power. **
awk reads from the standard input file or input information, and as sed, the information is read line by line read. The difference is, awk command line of the text file as a record, but a certain part of the row (column) as a field of the record. To operate these different fields (columns), a method similar to the position of the shell variables awk borrowed by $ 1, $ 2 ... $ 9 represents a procedure of a different column, a whole line of $ 0. Different fields and different fields may be separated by a specified manner, awk default delimiter is a space. awk command allows the use form "-F separator" to specify the delimiter.
awk command to / etc / passwd file processing of FIG:
awk contains several special built-in variables, such as:
2.awk command usage examples
1) row output text
[root@localhost ~]# awk '{print}' test.txt //输出所有内容，等同于“cat test.txt” [root@localhost ~]# awk '{print $0}' test.txt //输出所有内容，等同于“cat test.txt” [root@localhost ~]# awk 'NR==1,NR==3{print}' test.txt //输出第1~3行的内容 [root@localhost ~]# awk '(NR>=1) && (NR<=3) {print}' test.txt //输出第1~3行的内容 [root@localhost ~]# awk 'NR==1 || NR==3{print}' test.txt //输出第1行、第3行的内容 [root@localhost ~]# awk '(NR%2)==1 {print}' test.txt //输出所有奇数行的内容 [root@localhost ~]# awk '(NR%2)==0 {print}' test.txt //输出所有偶数行的内容 [root@localhost ~]# awk '/^root/{print}' /etc/passwd //输出以“root”开头的行 [root@localhost ~]# awk '/nologin$/{print}' /etc/passwd //输出以“nologin”结尾的行 [root@localhost ~]# awk 'BEGIN {x=0} ;/\/bin\/bash$/{x++};END {print x}' /etc/passwd //统计以/bin/bash结尾的行数 [root@localhost ~]# grep -c "/bin/bash$" /etc/passwd //统计以/bin/bash结尾的行数 [root@localhost ~]# awk 'BEGIN{RS=""}; END{print NR}' /etc/squid/squid.conf //统计以空格分隔的文件段落数
Note: When more command, use the "BEGIN ...... END"
2) Press the output text field
[root@localhost ~]# awk '{print $3}' test.txt //输出每行中（以空格分隔）的第3个字段 [root@localhost ~]# awk '{print $1,$3}' test.txt //输出每行中（以空格分隔）的第1个和第3个字段 [root@localhost ~]# awk -F ":" '$2==""{print}' /etc/shadow //输出/etc/shadow文件中（以“：”分隔）的第二个字段（密码为空的用户） [root@localhost ~]# awk 'BEGIN {FS=":"}; $2=""{print}' /etc/shadow //输出/etc/shadow文件中（以“：”分隔）的第二个字段（密码为空的用户） [root@localhost ~]# awk -F ":" '$7~"/bash"{print $1}' /etc/passwd //输出以“：”分隔且第7个字段中包含“/bash”的行的第1个字段 [root@localhost ~]# awk '($1~"nfs") && (NF==8) {print $1,$2}' /etc/services //输出包含8个字段且第1个字段中包含“nfs”的行的第1、2个字段 [root@localhost ~]# awk -F ":" '($7!="/bin/bash") && ($7!="/sbin/nologin") {print}' /etc/passwd //输出第7个字段既不为“/bin/bash”也不为“/bin/nologin”的所有行
3) through a pipe, call the Shell command double quotes
[root @ localhost ~] # awk -F: '/ the bash $ / Print {| "wc -l"}' / etc / the passwd
// call the number of the user "wc -l" command usage statistics "bash" of
[root ~ @localhost] -C # grep "the bash $" / etc / the passwd
// supra same effect a command
[root @ localhost ~] # awk 'BEGIN {while ( "w" | getline) n ++; {print n-2 }} '
// call the "w" command and force ah statistics online users
[root @ localhost ~] # awk' BEGIN { "hostname" | getline; Print $ 0} '
// call the "hostname" command, and the output current username
4) Use the awk command simple math
[root@localhost ~]# awk 'BEGIN{ a=6;b=3;print"(a + b)=",(a + b)}' (a + b)= 9 [root@localhost ~]# awk 'BEGIN{ a=6;b=3;print"(a - b)=",(a - b)}' (a - b)= 3 [root@localhost ~]# awk 'BEGIN{ a=6;b=3;print"(a / b)=",(a / b)}' (a / b)= 2 [root@localhost ~]# awk 'BEGIN{ a=6;b=3;print"(a % b)=",(a % b)}' (a % b)= 0