Three Musketeers, shell scripts

First, the Three Musketeers (sed, awk, grep)

(A) The Three Musketeers -sed

sed Definition: character stream editor: stream editor

sed functions and versions:

① plain text files, logs, configuration files, etc. == "linux

② add, delete, modify, query

③sed --version,GNU sed version 4.2.1

(1) sed syntax:

sed sed [options] [command] [input file]

sed -i.bak 's#oldboy#oldgirl#g' oldboy.txt

Sed command parameters: i.

s: sed command / instruction

g: - Small tail / modification

(2) sed execution:

① the first line read into memory (spatial mode)

② determine whether it is whether I want this line to meet the conditions (if not, re-read)

③ execution command inside sed s, d, i.

④sed command default display / output current memory, the default output - "to the screen

⑤ continue reading the next line until the last line of the file to read.

(3) common features of sed p (print)

查看某一行内容：
[root@oldboy-01 oldboy]# sed -n '1p' person.txt 
101,oldboy,CEO
[root@oldboy-01 oldboy]# sed  '1p' person.txt #如果不带参数n，显示全部内容
101,oldboy,CEO
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
查看2-4行内容：
[root@oldboy-01 oldboy]# sed -n '2,4p' person.txt 
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
查看包含oldboy的行：
[root@oldboy-01 oldboy]# sed -n '/oldboy/p' person.txt 
101,oldboy,CEO
查看包含oldboy至以104开头的行内容：
[root@oldboy-01 oldboy]# sed -n '/oldboy/,/^104/p' person.txt 
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO

(4) sed command parameters commonly -r

-r sed options, support for extended regular expression (|, ())

By default, sed only supports basic regular expressions.

[root@oldboy-01 oldboy]# egrep 'oldboy|yy' person.txt 
101,oldboy,CEO
104,yy,CFO
[root@oldboy-01 oldboy]# sed -rn '/oldboy|yy/p' person.txt 
101,oldboy,CEO
104,yy,CFO
[root@oldboy-01 oldboy]# 

Summary: ① a row query, the query certain range: 1p, 2,4p

② regular expression search any

sed -n '/oldboy/,/^104/p' person.txt

③sed command a regular expression to filter through, equivalent to egrep

sed show discontinuous line:

[root@oldboy-01 oldboy]# sed -n '1p;3p;4p' person.txt 
101,oldboy,CEO
103,Alex,COO
104,yy,CFO
[root@oldboy-01 oldboy]# 

(5) sed common features of increasing a (append) and i (insert) inserted

a: adding one or more lines of text in the specified line.

i: adding a line which multiple lines of text before the specified line.

[root@oldboy-01 oldboy]# sed '2a 106,xiaoyu,cxo' person.txt 
101,oldboy,CEO
102,zhangyao,CTO
106,xiaoyu,cxo
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboy-01 oldboy]# sed '2i 107,zhangsan,xxo' person.txt 
101,oldboy,CEO
107,zhangsan,xxo
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboy-01 oldboy]#

$ sed in the last line:

Increased after the last line of the multi-line method of increasing the use :( 1 / n line feed)

[root@oldboy-01 oldboy]# sed '$a new,new,new/nold,old,old' person.txt 
101,oldboy,CEO
102,zhangyao,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
new,new,new/nold,old,old
[root@oldboy-01 oldboy]# 

Method 2 content increased after the last line:

[root@oldboy-01 oldboy]# cat>>person.txt<<EOF

(6) common features of sed to delete and remove blank lines

-d: sed delete command parameters

删除一行：
[root@oldboy-01 oldboy]# sed '$d' person.txt 
101,oldboy,CEO
102,zhangyao,CTO
106,xiaoyu,cxo
103,Alex,COO
104,yy,CFO
删除多行：
[root@oldboy-01 oldboy]# sed '2,3d' person.txt 
101,oldboy,CEO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboy-01 oldboy]# 

Delete blank lines:

sed '/^$/d' person.txt

Show empty rows:

sed -n '/^$/p' person.txt

Display non-empty row:

sed -n '/^$/!p' person.txt

To delete the last line:

sed '$d' person.txt

Delete the last line of non-:

sed '$!d' person.txt

Display non-blank line (awk):

awk '!/^$/' person.txt

Alternatively common functions at the reference variables (7) sed

s: used alone, the string in each row of the first match to be replaced.

g: global replace all of each line, one replacement flag sed s instructions (global replacement)

[root@oldboy-01 oldboy]# sed 's#o#AAA#g' person.txt 

Modify the file parameters: -i

Automatic backup:

-i.ori: represents the first backup source files, modify the contents of the file.

Variable substitution:

Single quote: WYSIWYG

Double quotes: special symbol will be resolved.

[root@oldboy-01 oldboy]# x=oldboy
[root@oldboy-01 oldboy]# y=oldgirl
[root@oldboy-01 oldboy]# sed "s#$x#$y#g" person.txt 
101,oldgirl,CEO
102,zhangyao,CTO
106,xiaoyu,cxo
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@oldboy-01 oldboy]#

Direction Quote:

(): The function can remember part of a regular expression.

Extended regular, use the -r

\ 1 matches a reference first small brackets, \ 2 refers to the second small brackets matches, sed can remember up to 9.

Case: echo 'i am oldboy teacher', if you want to retain this line word oldboy.

(B) The Three Musketeers -awk

(1) awk basis

awk version:

[root@oldboy-01 ~]# awk --version

GNU Awk 3.1.7

awk examples:

[root@oldboy-01 ~]# awk -F ':' 'NR==2{print $1,$2}' /etc/passwd
bin x

awk formats:

awk parameters 'operation mode {}' file

awk parameters 'condition (whom) {doing}' file

awk implementation process:

① reads one line

② mode (condition) is the line I have to deal with it?

Is: read the line in order to perform the action

No: Repeat the process until the end of the last file.

④ read the next line

⑤end module - end

Pattern matching: Mode and Action

Case ①: display xiaoyu last name and id number (outside braces are conditions):

[root@oldboy-01 files]# awk '/Xiaoyu/{print $1,$2,$3}' reg.txt 
Zhang Xiaoyu 390320151

The second column contains xiaoyu: the wavy line are matched or comprises means

[root@oldboy-01 files]# awk '$2~/Xiaoyu/' reg.txt 
Zhang Xiaoyu    390320151  :155:90:201
[root@oldboy-01 files]# 


[root@oldboy-01 files]# awk '/Xiaoyu/' reg.txt #任意一列包含
Zhang Xiaoyu    390320151  :155:90:201
[root@oldboy-01 files]# awk '$0~/Xiaoyu/' reg.txt #$0,表示这一行。
Zhang Xiaoyu    390320151  :155:90:201

Case ②: Display all human id number beginning with 41 full name and id number:

[root@oldboy-01 files]# awk '$3~/^41/' reg.txt 
Zhang Dandan    41117397   :250:100:175
Liu   Bingbing  41117483   :250:100:175
[root@oldboy-01 files]# awk '$3~/^41/{print $1,$2,$3}' reg.txt 
Zhang Dandan 41117397
Liu Bingbing 41117483
[root@oldboy-01 files]# 

Case ③: Display all last digit ID number is 1 or 5 full name

[root@oldboy-01 files]# awk '$3~/[15]$/' reg.txt 
Zhang Xiaoyu    390320151  :155:90:201
Wu    Waiwai    70271111   :250:80:75
Wang  Xiaoai    3515064655 :50:95:135
Li    Youjiu    918391635  :175:75:300
Lao   Nanhai    918391635  :250:100:175
[root@oldboy-01 files]# awk '$3~/[15]$/{print $1,$2}' reg.txt 
Zhang Xiaoyu
Wu Waiwai
Wang Xiaoai
Li Youjiu
Lao Nanhai
[root@oldboy-01 files]# awk '$3~/(1|5)$/{print $1,$2}' reg.txt 
Zhang Xiaoyu
Wu Waiwai
Wang Xiaoai
Li Youjiu
Lao Nanhai
[root@oldboy-01 files]# 

Case ④: display xiaoyu contributions, when each value is to start with a $, such as ¥ 520

awk replace Usage: gsub (/ target /, "replaced by nothing", the first of several columns)

[root@oldboy-01 files]# awk '{gsub(/:/,"$",$4);print}' reg.txt 
Zhang Dandan 41117397 $250$100$175
Zhang Xiaoyu 390320151 $155$90$201
Meng Feixue 80042789 $250$60$50
Wu Waiwai 70271111 $250$80$75
Liu Bingbing 41117483 $250$100$175
Wang Xiaoai 3515064655 $50$95$135
Zi Gege 1986787350 $250$168$200
Li Youjiu 918391635 $175$75$300
Lao Nanhai 918391635 $250$100$175
[root@oldboy-01 files]# awk '$2~/Xiaoyu/{gsub(/:/,"$",$4);print}' reg.txt 
Zhang Xiaoyu 390320151 $155$90$201
[root@oldboy-01 files]# 

(2) the special mode awk BEGIN and END:

BEGIN {} BEGIN inside the content, awk run before reading the content.

Testing, calculation.

[root@oldboy-01 files]# awk 'BEGIN{print "this is kt"}'
this is kt
[root@oldboy-01 files]# awk 'BEGIN{print "this is kt"} {print NR,$0}' reg.txt 
this is kt
1 Zhang Dandan    41117397   :250:100:175
2 Zhang Xiaoyu    390320151  :155:90:201
3 Meng  Feixue    80042789   :250:60:50
4 Wu    Waiwai    70271111   :250:80:75
5 Liu   Bingbing  41117483   :250:100:175
6 Wang  Xiaoai    3515064655 :50:95:135
7 Zi    Gege      1986787350 :250:168:200
8 Li    Youjiu    918391635  :175:75:300
9 Lao   Nanhai    918391635  :250:100:175
[root@oldboy-01 files]# 

END {}: END {} inside the content, after a run awk scanned and the last line.

To display the final results. First computing, end displays the results.

Case ①: the number of blank lines statistics / etc / services file inside.

i = i + 1, i ++ generally used for statistical.

[root@oldboy-01 files]# awk '/^$/{i=i+1;print i}' /etc/services 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@oldboy-01 files]# awk '/^$/{i=i+1}END{print i}' /etc/services 
16

case study:

(Baidu Interview) for the following file content, the domain name is removed and counted the sorting process according to the domain name.

awk array Statistics: The hotel [110] = 'zhangsan'

[root@oldboy-01 files]# awk 'BEGIN{h[110]="lee";h[114]="xo";print h[110],h[114]}'
lee xo

Create a scene:

[root@oldboy-01 files]# cat>> url.txt << EOF
> http://www.etiantian.org/index.html
> http://www.etiantian.org/1.html
> http://post.etiantian.org/index.html
> http://mp3.etiantian.org/index.html
> http://www.etiantian.org/3.html
> http://post.etiantian.org/2.html
> EOF

Case:

[root@oldboy-01 files]# awk -F "[/.]+" '{print $2}' url.txt 
www
www
post
mp3
www
post
[root@oldboy-01 files]# awk -F "[/.]+" '{h[$2]=h[$2]+1;print h["www"]}' url.txt 
1
2
2
2
3
3
[root@oldboy-01 files]# awk -F "[/.]+" '{h[$2]=h[$2]+1;print h["www"]}' url.txt 
1
2
2
2
3
3
[root@oldboy-01 files]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{print h["www"],h["post"],h["mp3"]}' ur.txt 
3 2 1
[root@oldboy-01 files]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{for(x in h) print x}' url.txt 
www
mp3
post
[root@oldboy-01 files]# awk -F "[/.]+" '{h[$2]=h[$2]+1}END{for(x in h) print x ,h[x]}' url.txt 
www 3
mp3 1
post 2

Exercises: Statistical analysis secure system files. Who crack your password (the number of failed password each ip address appears)

The number of times each user analysis system to break.

(3) Summary:

①awk number 组进 line statistics, i ++, i = i + 1; i = i + $ n

②awk execution

By regular pattern as ③awk

④BEGIN special mode and END

Two, shell programming

(A) shell basics

Command interpreter: / bin / bash

file view the file type, whether it is a shell script.

① unified on the script / server / scripts

② recommended vim editor

③ script the first line:! (# / Bin / bash, bash is the default interpreter # des, # author:., # Time, # version :)

Variable: See name EENOW (beginning with the letter, letters, numbers, underscores)

Take variable content: such as: $ {week}, using braces reference variables.

Ordinary variables: local variables

Environment variables: global variables, capital of PATH, LANG, PS

Local variables become global variables: export variable

Use env, view the global variables.

Cancel variables; unset variables

Environment variables associated with files and directories

Global environment variable configuration file:

/etc/profile

/etc/bashrc

/etc/profile.d/

Users log on to the system will run the script directory

Scripts have execute permissions.

User environment variable configuration file:

~/.bash_profile

~/.bashrc

(B) special variables

Position variable: $: 0, n, #:

0: script name, including the path.

n: n-th argument.

($ #) Indicates how many parameters, the total number of parameters.

Process state variables: (? $) Is displayed on a command execution result.

Command executed correctly: 0

Command execution error: Non 0

(C) shell programming method of variable assignment: read

Interactive assignment:

格式：read -p "input x y:" x y

[root@oldboy-01 files]# cat  xxyy.sh
#!/bin/bash
read -p "input x y:" x y
echo $x $y
[root@oldboy-01 files]# sh xxyy.sh
input x y:10 50
10 50
[root@oldboy-01 files]# 

(D) shell script loop

Conditional Expression: Format: [<test expression>]

Analyzing files: f, d

[-F /root/oldboy.alex.txt] whether to file

[-D / root] directory exists

Judgment integer:

Equal to equal: -eq

Does not equal not equal: -ne

Greater than greater than: -gt

Greater than or equal greater equal: -ge

Less than less than: -lt

Less less equal: -le

Simple Case: determining the number of command-line parameter is equal to 2

[root@oldboy-01 files]# cat /server/scripts/arg.sh
#!/bin/bash
[ $# -eq 2 ] && echo "arg:" $#
[root@oldboy-01 files]# sh /server/scripts/arg.sh a b
arg: 2
[root@oldboy-01 files]# sh /server/scripts/arg.sh a b c
[root@oldboy-01 files]# 

Case: If / oldboy directory does not exist, create

[ -d /oldboy ] || mkdir -p /oldboy

If there is then prompted /root/oldboy.txt file already exists.

[ -f /root/oldboy.txt ] && echo "file esists"

determining if programming of shell: if elif, else, fi:

[root@oldboy-01 files]# sh /server/scripts/comp.sh  1 1
1 equal 1
[root@oldboy-01 files]# sh /server/scripts/comp.sh  1 4
1 less than 4
[root@oldboy-01 files]# sh /server/scripts/comp.sh  1 0
1 greater than 0
[root@oldboy-01 files]# cat /server/scripts/comp.sh 
#!/bin/bash
num1=$1
num2=$2
if [ $# -ne 2 ];then
   echo "usage:please input 2 number:num1 num2"
   exit
fi

if [ $num1 -gt $num2 ];then
   echo $num1 greater than $num2
elif [ $num1 -lt $num2 ];then
   echo $num1 less than $num2
else
   echo $num1 equal  $num2
fi
[root@oldboy-01 files]# 

Analyzing the shell programming expression summary conditions (first draw flowcharts)

1, the conditional expression

Integer comparison: -eq, -gt, -ge, -ne, -lt, -le

File Compare:

[ -d /oldboy ]

[-f /oldboy/oldboy.txt ]

2, is determined

if ,elif,else,fi

The shell programming for loop :( do the same thing)

Format: for variable names in the list

do

command

done

[root@oldboy-01 files]# for num in {01..100}
> do
>    echo "the $num number is :$num"
> done
the 001 number is :001

Case: linux boot optimization project, leaving only crond, sshd, network, rsyslog, systat, others are closed.

chkconfig service off

[root@oldboy-01 scripts]# sh  chk.sh
[root@oldboy-01 scripts]# chkconfig | grep 3:on
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
[root@oldboy-01 scripts]# cat chk.sh
#!/bin/bash
for name in $(chkconfig | egrep "crond|sshd|rsyslog|network|systat" -v | awk '{print $1}')
do
    chkconfig $name off
done
[root@oldboy-01 scripts]#

(E) linux to run the script tips:

Look at the process script: -x: Yes, plus running process, no plus sign to display the desktop.

① special variables: ($ n, 0, # ,?)

② ordinary variable naming rules

③ the difference between ordinary and environment variables

④ files and directories related to user

⑤ conditional expression determines, loop

⑥ optimize system startup project

⑦ bulk add users and set random passwords

Third, the network

Generating a random source port protocol, in the range of:

[root@oldboy-01 ipv4]# cat  /proc/sys/net/ipv4/ip_local_port_range 

4000    65000

The domain name hierarchy levels:

The first level: called the root domain name server (13)

The second level: called top-level domain name server (a domain name server) com, cn, gov

The third level: called secondary domain name server (DNS server authorization) .baidu.com

Fourth grade: called a virtual machine host information www.baidu.com

Use the dig command to check the name resolution process.

nslookup DNS

host domain name resolution

ping DNS

DNS configuration places:

/etc/sysconfig/network-scripts/ifc-eth0

/etc/resolv.conf

Restart card:

ifdown eth0 && ifup eth0

/etc/init.d/network restart

Default routing:

route -n
route add default gw 10.0.0.254
route del default gw 10.0.0.254
route add -net 172.16.1.0/24 gw 10.0.0.254
route add -host 172.16.2.2 gw 10.0.0.254

Multi-NIC address configuration:

方法1：
ifconfig eth0:1 10.1.0.8/24 up
ifconfig eth0:1 down
方法2
ip addr add 10.1.0.8/24 broadcast 10.1.0.255 dev eth0

Network commonly used commands:

ifconfig
ip addr
route
hostname
hostname -I
netstat
ss -lntp | grep sshd(centos7)
netstat -lntp | grep sshd
ss -lntp | grep sshd
ping -c 4 10.0.0.1
[root@oldboy-01 ipv4]# ping 10.0.0.1 -c 4 -i 3
(ping 4次，间隔3s)
[root@oldboy-01 ipv4]# ping 10.0.0.1 -c 4 -i 3 -q #不显示过程
[root@oldboy-01 ipv4]# ping 10.0.0.1 -f -c 1000
（-f 极限ping测，很快出结果）
tracroute www.baidu.com
tracert www.baidu.com
telnet 22 #测试端口
[root@oldboy-01 ipv4]# nmap 10.0.0.202 -p 10-40