-n : display line number

-o : only show matches

-i : do not distinguish uppercase or lowercase letters

-v : negate

-E : support extensions

-q : execute silently

-w : recognize as word

-c : Count the number of matching lines

2) Metacharacters of regular expressions

^ : matches the beginning of the line

$ : matches the end of the line

. : Match any single character

.* : match any character

[] : match the content inside the brackets

[-] : Match the range of content inside the brackets

{n} : the number of times the range matched

[n1-n2] : match start and end content

{ n,m } : Repeat the previous character n to m times

{ n, } : Repeat the preceding character at least n times

[^] : matches any character outside the brackets

\ : escape character

\< : fixed prefix

\> : fixed ending

3. The basic application of regular expressions (grep—————one of the three cheap customers)

1) The filter keyword is the displayed

grep -n 'the' test.txt

2) View the beginning of the line is the displayed

grep -n '^the' test.txt

3) Match any single character

grep -n 'w.d' test.txt

4) Only show matching content

grep -o 'wod' test.txt

5) Wildcard application matching begins with w and ends with any character in the middle of d

grep -n 'w*d' test.txt

6) Match the content inside the brackets ( the brackets mean "or" )

grep -n 'sh[io]rt' test.txt

7) Basic application of diverted characters, filtering IP address information

grep -n -E "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" test.txt

Equivalent to

egrep -n "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" test.txt

4. Extended regular expressions

1) Extended regular expressions represent common metacharacters

+ : Repeat one or more times before the character is displayed

? : Display the character before the current letter

| : Alternatively, display multiple characters

() : lookup group string

()+ : identify multiple repeating groups

2) Application of extended regular expressions

egrep -n 'wo+d' test.txt	Display the content of repeated o
egrep -n '?d' test.txt	Display the content in front of d
egrep -n 'The\|the' test.txt	Display the matching The or the
egrep -n '(h\|o\|r)' test.txt	Display the content of the line containing the characters h, o, r
egrep -n 's(hor)t' test.txt	Display the words that start with s and end with t and are continuous in the middle of hor

According to the above test:

egrep -n 'wo+d' test.txt

egrep -n '?d' test.txt

egrep -n 'The|the' test.txt

egrep -n '(h|o|r)' test.txt

egrep -n 's(hor)t' test.txt

Second, the use of sed The second brother—————————————————————————————————————————————————————————————————————————————————————————————————— sed

1. The role of sed and the workflow of sed

1) The role of sed

sed is a text file analysis conversion tool

Used to read, modify, and display data

2) The function process of sed

Read data: read data from files, input, pipeline commands, store data temporarily

Execution: Non-interactive modification of data in the pattern manipulation space according to the sed command

Display: print and display the modification results of the text file

2. Common options and operators of sed command

1) sed command options

-n : display the processed results

-h : show help

-e : Specifies the command or script to process the input text file

-f : Specifies that the script process the input text file

-i: edit the text file directly

2) The operation symbol of sed

a : add content

d : delete content

p : print content

s : replace content

y : character conversion

c : replace a specific line with the specified content

3. The basic application of sed

1) Display 1 to 5 rows of data

sed -n '1,5p' test.txt

2) Display odd lines

sed -n -e '1p;3p;5p' test.txt

3) Delete the first row of data

sed '1d' test.txt

4) Remove empty lines

sed -i '/^$/d' a.sh

delete comment

sed -i ‘/^#/d’ a.sh

5) Change uppercase THE to lowercase the

sed -i 's/the/The/' test.txt

6) He keyword to add comments

sed -i '/^ip/s/^/#/' test.txt

7) Migrate characters to migrate lines 1~5 to line 15

sed -i '1,3{H;d};8G' test.txt

8) The output only shows the sed usage of ip information

ifconfig ens3 3 | sed –n '2 s/.* in a //p' | sed 's/n.*//'

ifconfig ens33 |sed -nr '2 s/.*et (.*) n.*/\1/gp'

3. The use of awk The third brother —————————awk

1. The role of awk

Filter and read data using

2. Variables of awk

（-F）

FS: The delimiter of the text field, spaces can be used as placeholders

NF: number of rows processed

NR: row number of processed data

$0: entire row of data to process

$n: the column data of the processed data row

3. The basic application of awk

1) Show all content

awk '{print}' 1.txt

2) Display the first column of data

awk '{print $1}' 1.txt

3) Between the first and second column add --

awk '{print $1"--"$2}' 1.txt

4) Display the first and second row of data

awk 'NR==1,NR==2{print}' 1.txt

expand:

1. Sorting and checking for duplicates of historical commands, statistics

Example: history

history | sort

history | sort –nr sorting r is reverse order

history | awk '{print $2}'

sort | uniq –c deduplication, count occurrences

history | awk '{print $2}' | sort |uniq -c |sort -nr |head -5

Count the number of commands used in history from largest to smallest and display the top five with the highest number of occurrences

history | awk '{print $2}' | sort |uniq -c |sort -nr |sed -n '1,5p'

history | awk '{print $2}' | sort |uniq -c |sort -nr |awk 'NR<=5{print}'

cat www.new1019.xianqujingpinwang.log| sed -n '/23\/Nov\/2022:0/,/23\/Nov\/2022:12/p'|grep `date "+%d/%b"`|awk '{print $1}'|sort |uniq -c |sort -rn|wc -l

View the log files of accessing the current website, and filter out the ip address with the highest frequency within 12 hours, in case this ip address is a malicious attack ip

The above experiment can be verified by our apache experiment instead!

cat /var/log/httpd/access_log |awk '{print $1}'

cat access_log |awk '{print $1}' |sort|uniq -c |sort -nr |wc -l

It can be counted that within the current time, only one IP address has visited our httpd

Regular expressions and "three cheap customers"

Basic applications of shell scripts :

1. Regular expressions and grep

2) Composition of regular expressions

1) Common options for regular expression grep