Shell programming (3) grep sed awk text processing three musketeers

Previous chapter:

Shell programming (2)_Do test meow sauce blog-CSDN blog

1. ps command

Command:  ps
Function:  Mainly to view the process information of the server
Options meaning:
-e: equivalent to '-A', which means to list all processes
-f: display all columns (display all fields) 

Two, grep (text filtering)

grep is mainly used for filtering. Filter text by line. If a line contains the keyword to be searched, the entire line will be output.

2.1 Basic usage of grep

View grep help documentation grep --help

[root@ecs-39233 chenshuai]# grep --help
Usage: grep [OPTION]... PATTERN [FILE]...
Search for PATTERN in each FILE or standard input.
PATTERN is, by default, a basic regular expression (BRE).
Example: grep -i 'hello world' menu.h main.c

Two forms of grep application

Form 1: Search for one file/multiple files.

grep [OPTION]... PATTERN [FILE1,FILE2 ... ]

[OPTION] Optional, you can fill in the following content 

PATTERN keyword, the keyword to be searched for. Searches from files FILE1 FILE2 for lines containing the keyword PATTERN.

 Form 2: When executing a command, search for data in the command result

some command | grep [option] pattern

[OPTION] Optional, you can fill in the following content 

PATTERN keyword, the keyword to be searched for. There can be no quotes, single quotes, or double quotes.

Without quotes, there must be no spaces in the PATTERN keyword. 

Single quotes, variables in the PATTERN keyword cannot be parsed, what is searched for is what

Double quotes, can resolve variables in the PATTERN keyword.

Pass the output of a command (command) to the following grep command through the pipe character |.

In the output of the first command, search for the keyword pattern

options	effect
-i	ignore case
-c	output only the number of matching lines
-n	show line number
-r	recursive search
-E	Support for extended regular expressions
-W	match whole word
-I	Only list matching filenames
-F	Regex is not supported, match the literal meaning of the string

Example 1

Search whether a user exists in the current Linux system (/etc/passwd).

Is there a root user

grep root /etc/passwd

[root@ecs-39233 ~]# grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
[root@ecs-39233 ~]# 

Is there a miao user

[root@ecs-39233 ~]# grep miao /etc/passwd
[root@ecs-39233 ~]# 

Example 2 

View the current system, java process information

ps -ef |grep java

View tomcat process information

ps -ef |grep tomcat

Example 3   ignore case -i

ignore case -i

grep does not ignore case by default. grep -i ignores case.

don't ignore case

[root@ecs-39233 ~]# echo "Hello world" |grep hello
[root@ecs-39233 ~]# 

ignore case

[root@ecs-39233 ~]# echo "Hello world" |grep -i hello
Hello world

Example 3 Statistics total quantity-c

Statistics total quantity-c

Check out the Linux user documentation. /etc/passwd

[root@ecs-39233 ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

This kind of user ending in nologin is a user without login permission.

Count how many users in /etc/passwd do not have login permissions. -c

[root@ecs-39233 ~]# grep -c "/sbin/nologin" /etc/passwd
16

Example 4 Search multiple files

1. Search multiple files

In the two files test_while.sh test_read_n.sh, search for "/bin/bash" string

[root@ecs-39233 ~]# grep "/bin/bash" test_while.sh test_read_n.sh
test_while.sh:#!/bin/bash
test_read_n.sh:#!/bin/bash

2. Filter the directory -r

I want to search all files under the /chenshuai directory. Search for "/bin/bash" string

grep -r "/bin/bash" /chenshuai

[root@ecs-39233 chenshuai]# grep -r "/bin/bash" /chenshuai
/chenshuai/study_shell.sh:#!/bin/bash
/chenshuai/until.sh:#!/bin/bash
/chenshuai/miao_test.sh:#!/bin/bash

3. When filtering, only print the file name. Do not print hit string -l

grep -rl "/bin/bash" /chenshuai

/chenshuai/study_shell.sh
/chenshuai/until.sh
/chenshuai/miao_test.sh

Example 5 Filter the data before and after the target row

Display the last few rows of data of the target row-A

Display the first few rows of data of the target row -B

Display several rows of data before and after the target row -C

View grep help documentation

grep --help

See the grep help documentation for the usage of -A. ("\-A" escape)

grep --help | grep "\-A"

[root@ecs-39233 ~]# grep --help | grep "\-A"
  -A, --after-context=NUM   print NUM lines of trailing context

Display the last few lines of data of the target line (-A ,after)

View the -A line and the last 3 lines of data.

grep --help | grep -A3 "\-A"

 

View - B this row, and the data of the first 4 rows.

Display several rows of data before and after the target row -C

View the data of 2 rows before and after the current row

grep --help|grep -C2 "\-C"

2.2 grep + regular

2.2.1 Basic regularization

^ 以x开头

$ 以x结尾

.

*

[]

[^]

Search in files for strings starting with "r"

grep '^r' /etc/passwd

2.2.2 Extended regular grep -E or egrep

Extended regex: (commonly used)

 {}

()

|

+

?

In the file, match the phone number

grep -E '^1[3456789]\d{9}$' /etc/xxx.log

egrep '^1[3456789]\d{9}$' /etc/xxx.log

2.2.3 Filter multiple keywords 

I want to search for data in the file that contains multiple keywords -E -A -B (these keywords are in the relationship of OR)

Notice:

To filter multiple keywords, you need to use the pipe symbol | in the regular expression to represent the relationship of or

 The pipe character | in the regular rule belongs to the extended regular rule, and needs to use grep -E or egrep

-E requires diversion\-E

grep --help | egrep '\-E|\-A\-B'

grep --help | grep -E '\-E|\-A|\-B'

 2.3 grep for writing shell scripts

Realize the function:

A shell script to check the number of cpu cores.

Small expansion of Linux knowledge:

Use the top command to view Linux information. Then press 1, and all cpu usage information will be displayed at the top.

View the configuration information of the cpu,

cat /proc/cpuinfo

Inside, each core has a model name, so counting the number of cpu cores can be achieved by counting the number of model name texts.

grep -c "model name" /proc/cpuinfo

3. sed (Add, delete, modify and check files)

The full name of sed is stream  editor, stream editor. Edit text in a non-interactive way.

The sed editor is not destructive, it will not modify files unless shell  redirection is used to preserve output. By default, all output lines are printed to the screen.

 sed working process:
The sed editor processes the file (or input) line by line and sends the output to the screen. sed's commands are those found in vi and ed/ex editors. sed keeps the line it is currently processing in a temporary buffer called the pattern space or temporary buffer. When sed finishes processing a line in the pattern space (ie, executes a sed command on that line), it sends the newline to the screen (unless there was a previous command to delete the line or cancel the print operation). sed terminates after processing the last line of the input file. sed stores each line in a temporary buffer, and edits this copy, so the source file is not modified or corrupted.
 

effect:

The sed stream editor can add, delete, modify and check files.

We process files on a daily basis and use vim commands. But in a shell script, vim cannot be used to process text, vim is an interactive command, we can use the sed command.

How to handle text:

Process text by line, similar to grep.

Two forms:

Method 1: Process the file

sed [option] "pattern command" file

[option] Optional extension items, such as -n -f -r, etc.

pattern The keyword to search for. If pattern is empty, the default is to process each row.

What does cmond do with the hit line of data. (processing command)

file file path

When sed prints, the original data will be printed by default.

Method 2: Process the output of the command

some command | sed [option] "pattern command"

 Pass the output of a command command to sed for processing.

In this result, sed looks for a matching line, and then processes this matching line.

options	effect
-n	print only lines that match the pattern
-f	Load the file that holds the action
-r	Support extended regular
-i	Support for modifying files

3.1 Printing files

sed 'p' /etc/passwd

Without a search term, each row is processed by default.

'p' print command, shorthand for print.

[root@ecs-39233 ~]# sed 'p' /etc/passwd
root:x:0:0:root:/root:/bin/bash
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

It is found that in the passwd file, each line of data is printed twice. Because sed prints the original data once and the hit data once, it prints it twice.

-n Print only lines matching the pattern. (only print 1 line)

sed -n 'p' /etc/passwd

[root@ecs-39233 ~]# sed -n 'p' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin

3.2 Four types of matching word pattern

match pattern	meaning
3	only process line 3
3,6	Only process lines 3 to 6
/pattern1/	Only process lines that match pattern1
/pattern1/,/pattern2/	Only process lines matching pattern1 to pattern2

command command 

Order	effect
query p	Print
add a	Abbreviation for add, add after matching line

3.2.1 Use line number to filter, pattern is line number

Method 1: operate on the file

sed -n "第几行 操作命令" 文件路径

pattern is a specific line number.

For example, print the content of line 3

sed -n '3 p' /etc/passwd

[root@ecs-39233 ~]# sed -n '3 p' /etc/passwd
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@ecs-39233 ~]# 

Method 2: Process the result of a command

cat /etc/passwd | sed -n '3 p'

For the running result of cat, print the data in line 3. 

[root@ecs-39233 ~]# cat /etc/passwd | sed -n '3 p'
daemon:x:2:2:daemon:/sbin:/sbin/nologin

Process consecutive lines 'x,y command'

For example, print the data from line 4 to line 8

sed -n '4,8 p' /etc/passwd

[root@ecs-39233 ~]# sed -n '4,8 p' /etc/passwd
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
[root@ecs-39233 ~]# 

3.2.2 Use regular filtering, pattern is a regular expression

expression:

sed -n '/正则表达式/ 操作命令' 文件路径 

For example, print the line containing root in the /etc/passwd file

sed -n '/root/ p' /etc/passwd

[root@ecs-39233 ~]# sed -n '/root/ p' /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

For all lines starting with s, print /^sp/

sed -n '/^s/ p' /etc/passwd

[root@ecs-39233 ~]# sed -n '/^s/ p' /etc/passwd
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin

encounter/want to use escape

For example, print a line containing /sbin/nologin, where / needs to be escaped

sed -n -e '/\/sbin\/nologin/ p' /etc/passwd

[root@ecs-39233 ~]# sed -n -e '/\/sbin\/nologin/ p' /etc/passwd
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin

 -r supports extended regular

sed only supports basic regular expressions by default, and you need to add -r for extended regular expressions

Example:

Print a line containing 2 o's, o{2}

sed -n -r '/o{2}/ p' /etc/passwd

[root@ecs-39233 ~]# sed -n -r '/o{2}/ p' /etc/passwd
root:x:0:0:root:/root:/bin/bash
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin

Multiple regularizations are used to filter data from row x to row y.

Note that the multiple regularizations here are not an or relationship, but a meaning from row x to row y.

For example, I want to filter the data starting from the line starting with adm to the line starting with mail

sed -n '/^adm/,/^mail/ p' /etc/passwd

[root@ecs-39233 ~]# sed -n '/^adm/,/^mail/ p' /etc/passwd
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

3.3 sed modify text

3.3.1 Write content a after the target line (false modification, modified memory)

stencil:

sed  '/正则表达式/ a 新增内容' 文件路径 

add a,append

The modified text will only take effect on the current output screen. Check the text content, it has not actually been modified.

Prepare a file in advance (such as Tengwang Pavilion Preface.txt)

vim 滕王阁序.txt

豫章故郡，洪都新府。星分翼轸，地接衡庐。襟三江而带五湖，控蛮荆而引瓯越。物华天宝，龙光射牛斗之墟；人杰地灵，徐孺下陈蕃之榻。雄州雾列，俊采星驰。台隍枕夷夏之交，宾主尽东南之美。都督阎公之雅望，棨戟遥临；宇文新州之懿范，襜帷暂驻。十旬休假，胜友如云；千里逢迎，高朋满座。腾蛟起凤，孟学士之词宗；紫电青霜，王将军之武库。家君作宰，路出名区；童子何知，躬逢胜饯。

时维九月，序属三秋。潦水尽而寒潭清，烟光凝而暮山紫。俨骖騑于上路，访风景于崇阿；临帝子之长洲，得天人之旧馆。层峦耸翠，上出重霄；飞阁流丹，下临无地。鹤汀凫渚，穷岛屿之萦回；桂殿兰宫，即冈峦之体势。

披绣闼，俯雕甍，山原旷其盈视，川泽纡其骇瞩。闾阎扑地，钟鸣鼎食之家；舸舰弥津，青雀黄龙之舳。云销雨霁，彩彻区明。落霞与孤鹜齐飞，秋水共长天一色。渔舟唱晚，响穷彭蠡之滨；雁阵惊寒，声断衡阳之浦。

遥襟甫畅，逸兴遄飞。爽籁发而清风生，纤歌凝而白云遏。睢园绿竹，气凌彭泽之樽；邺水朱华，光照临川之笔。四美具，二难并。穷睇眄于中天，极娱游于暇日。天高地迥，觉宇宙之无穷；兴尽悲来，识盈虚之有数。望长安于日下，目吴会于云间。地势极而南溟深，天柱高而北辰远。关山难越，谁悲失路之人？萍水相逢，尽是他乡之客。怀帝阍而不见，奉宣室以何年？

Insert content into the file.

Example: For the line containing " September of Time Dimension ", add a line of content later.

sed '/时维九月/ a 作者：王勃' /chenshuai/滕王阁序.txt

[root@ecs-39233 chenshuai]# sed '/时维九月/ a 作者：王勃' /chenshuai/滕王阁序.txt
豫章故郡，洪都新府。星分翼轸，地接衡庐。襟三江而带五湖，控蛮荆而引瓯越。物华天宝，龙光射牛斗之墟；人杰地灵，徐孺下陈蕃之榻。雄州雾列，俊采星驰。台隍枕夷夏之交，宾主尽东南之美。都督阎公之雅望，棨戟遥临；宇文新州之懿范，襜帷暂驻。十旬休假，胜友如云；千里逢迎，高朋满座。腾蛟起凤，孟学士之词宗；紫电青霜，王将军之武库。家君作宰，路出名区；童子何知，躬逢胜饯。

时维九月，序属三秋。潦水尽而寒潭清，烟光凝而暮山紫。俨骖騑于上路，访风景于崇阿；临帝子之长洲，得天人之旧馆。层峦耸翠，上出重霄；飞阁流丹，下临无地。鹤汀凫渚，穷岛屿之萦回；桂殿兰宫，即冈峦之体势。
作者：王勃

Use the cat command to view the text content, but it has not been modified. 

3.3.2 After the target line, directly add text content -ia

 stencil:

sed  -i '/正则表达式/ a 新增内容' 文件路径 

add a,append

actually modify the contents of the file.

Example:

Add a line of remarks after the line "Sunset and the Lonely Bird Fly Together"

sed -i '/落霞与孤鹜齐飞/ a 备注：xxx' /chenshuai/滕王阁序.txt

Looking at the text content, cat
really adds a note.

3.3.3 Range operations

Perform operations on the data from row x to row y

sed '/正则1匹配行/,/正则2匹配行/ a 新增备注' 文件路径 

Example:

For the row of Shiwei September, all the rows up to the row of embroidered door, each row will add a new piece of data after it.

sed '/时维九月/,/披绣闼/ a 新增备注zzzzz' /chenshuai/滕王阁序.txt 

3.3.4 Add content before the target line i

Add content to the front of a line

 stencil:

sed  '/正则表达式/ i 新增内容' 文件路径 

Example:

sed '3 i 在第3行前增加数据' /chenshuai/滕王阁序.txt

What is used here is not regular, but the line number (line 3)

3.4 Read the external file, add it to the target file as content

Read an external file, as content, to add to the target file.

 

四、awk