Text processing tools and regular expressions, SHELL scripting

First, the text editor VIM

● Command Mode -> insert mode

i insert, the input at cursor

I entered the first line of the current cursor line

a append, after the input of the cursor

A cursor at the end of the current input line

o Open a new line below the current cursor line

O Open a new line above the current cursor line

● Insert mode --- ESC -----> command mode

● Command Mode ----: ----> Extended Command Mode

● Extended Command Mode --- ESC, enter ----> command mode

1, first enter the editing interface command mode, press the o key switch to insert mode, and the cursor is switched to the current line of the next line start a new line.

2, into the first editing interface command mode, press the O key to switch to insert mode, and the cursor is switched to the current line on the first line by line on a separate line.

3, first enter the editing interface command mode, press the A key to switch to insert mode, and the cursor is switched to the current line end of the line.

 

4, enter the command mode: r plus the file name, file contents can be read in the current file.

5, command mode input:! Plus command may exit the text editing interface execute the command, enter: r! Adding the output of the command to execute the command reads the current cursor line.

6, the command input mode: set number may be displayed line number.

7, enter the command mode: 2,4d, means to delete the contents of the line 2-4,: 2,4y copy the contents of the line 2-4, in the line where the cursor Press p to paste the contents. (P attached to the current row after row, P attached to the line when the line)

8, the content search and replace, enter the command mode, the command line for the root search 1-5, all of the root line 1-5 replaced admin, / g represents replace all, if not the first to change the default. % Represents full text

 

 9, vim advanced customization tools needed to create a home directory vimrc file

 

 Profiles:

/ Etc / vimrc (Global)

~ / .Vimrc (personal)

(1) display line numbers

Show: set number, abbreviated set nu

Cancel Display: set nonumber, abbreviated set nonu

Case (2) ignore characters

Enable: set ignorecase, abbreviated set ic

Do not ignore: set noic

(3) auto-indentation

Enable: set autoindent, abbreviated set ai

Disable: set noai

(4) Copy preserve formatting

Enable: set paste

Disable: set nopaste

(5) Display Tab and newline

Enable: set list

Disable: set nolist

(6) highlighted search

Enable: set hlsearch

Disable: set nohlsearch

(7) provided in the cursor line identification line

Enable: set cursorline, abbreviated set cul

Disable: set nocursorline

Second, common text processing tools

 1, cat -A show all control characters

 

2, cat -n exhibited on each line are numbered

 

3, tail -f trace file name, file change monitoring

 

 Generating a new terminal contact log

 cut command to extract the text in columns

command and writes the cut line cut from each byte, character, and these files byte field, and character fields to standard output.

If you do not specify the File parameter, cut command reads standard input. You must specify -b, -c or one of the -f flag.

• -b: divided in units of bytes. These byte positions will ignore multi-byte character boundary, unless you also specify the -n flag.
• -c: the character is divided into units.
• -d: Custom separator, default tabs.
• -f: Use with -d, which specifies the display area.
• -n: Cancel split multi-byte characters. Only used with the -b flag. If the last byte of a character is indicated by List parameter falls -b flag is
  in range, the character is written; otherwise, the character will be excluded

1, the first and third columns extraction etc / passwd file (cut -d divided by delimiter) 

 1 [22:49:16 root@Centos8 ~]#cut -d: -f1,3 /etc/passwd
 2 root:0
 3 bin:1
 4 daemon:2
 5 adm:3
 6 lp:4
 7 sync:5
 8 shutdown:6
 9 halt:7
10 mail:8
11 operator:11
12 games:12
13 ftp:14
14 nobody:65534
15 dbus:81
16 systemd-coredump:999
17 systemd-resolve:193
18 tss:59
19 polkitd:998
20 unbound:997
21 sssd:996
22 sshd:74
23 gjz:1000
24 postfix:89

 2、将df输出中的空格替换成%，再cut删除%号，取出第五列

 1 [22:49:44 root@Centos8 ~]#df | tr -s ' ' '%' |cut -d% -f5
 2 Use
 3 0
 4 0
 5 1
 6 0
 7 3
 8 1
 9 13
10 0

 

 wc命令收集文本统计数据

 利用wc指令我们可以计算文件的Byte数、字数、或是列数，若不指定文件名称、或是所给予的文件名为”-“，则wc指令会从标准输入设备读取数据。

• -c或–bytes或–chars 只显示Bytes数。
• -l或–lines 只显示列数。
• -w或–words 只显示字数。
• –help 在线帮助。
• –version 显示版本信息。

1 [22:58:53 root@Centos8 ~]#wc nianling.txt 
2  3  3 37 nianling.txt                     分别显示行数、单词数、字节数

 

1 [22:59:00 root@Centos8 ~]#wc -l nianling.txt 
2 3 nianling.txt                            只显示文件行数

sort命令文本排序

Linux sort命令用于将文本文件内容加以排序。

sort可针对文本文件的内容，以行为单位来排序。

• -b 忽略每行前面开始出的空格字符。
• -c 检查文件是否已经按照顺序排序。
• -d 排序时，处理英文字母、数字及空格字符外，忽略其他的字符。
• -f 排序时，将小写字母视为大写字母。
• -i 排序时，除了040至176之间的ASCII字符外，忽略其他的字符。
• -m 将几个排序好的文件进行合并。
• -M 将前面3个字母依照月份的缩写进行排序。
• -n 依照数值的大小排序。
• -o<输出文件> 将排序后的结果存入指定的文件。
• -r 以相反的顺序来排序。
• -t<分隔字符> 指定排序时所用的栏位分隔字符。
• +<起始栏位>-<结束栏位> 以指定的栏位来排序，范围由起始栏位到结束栏位的前一栏位。
• –help 显示帮助。
• –version 显示版本信息。

1、将etc/passwd文件的第3列排序，：作为分隔符，默认按照字符排序，加上n按照数字排序

 1 [23:07:31 root@Centos8 ~]#sort -nt: -k3 /etc/passwd
 2 root:x:0:0:root:/root:/bin/bash
 3 bin:x:1:1:bin:/bin:/sbin/nologin
 4 daemon:x:2:2:daemon:/sbin:/sbin/nologin
 5 adm:x:3:4:adm:/var/adm:/sbin/nologin
 6 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
 7 sync:x:5:0:sync:/sbin:/bin/sync
 8 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
 9 halt:x:7:0:halt:/sbin:/sbin/halt
10 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
11 operator:x:11:0:operator:/root:/sbin/nologin
12 games:x:12:100:games:/usr/games:/sbin/nologin
13 ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
14 tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
15 sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
16 dbus:x:81:81:System message bus:/:/sbin/nologin
17 postfix:x:89:89::/var/spool/postfix:/sbin/nologin
18 systemd-resolve:x:193:193:systemd Resolver:/:/sbin/nologin
19 sssd:x:996:993:User for sssd:/:/sbin/nologin
20 unbound:x:997:995:Unbound DNS resolver:/etc/unbound:/sbin/nologin
21 polkitd:x:998:996:User for polkitd:/:/sbin/nologin
22 systemd-coredump:x:999:997:systemd Core Dumper:/:/sbin/nologin
23 gjz:x:1000:1000:gjz:/home/gjz:/bin/bash
24 nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin

 2、找到分区利用率最大的值

[23:07:40 root@Centos8 ~]#df |tr -s ' ' '%'|cut -d% -f5|sort -nr|head -1
13

 3、统计访问次数最多的IP地址

以空格为分隔符列出第一列，sort排序，uniq  -c显示每行重复的次数，sort -nr倒序排序

 

1 [root@centos7 ~]#cut -d ' ' -f1 /root/access_log |sort |uniq -c|sort -nr
2    4870 172.20.116.228
3    3429 172.20.116.208
4    2834 172.20.0.222
5    2613 172.20.112.14
6    2267 172.20.0.227

 

 正则表达式

1、显示etc/passwd文件包含root的行

[root@centos7 ~]#grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

 2、以/dev/sd为字符列出磁盘分区并找出最大利用率值

1 [root@centos7 ~]#df |grep /dev/sd |tr -s ' ' %|cut -d% -f5 |sort -nr |head -n1
2 18

 3、取出etc/passwd文件中不包含/bin/bash的行

 1 [root@centos7 ~]#grep -v '/bin/bash' /etc/passwd
 2 bin:x:1:1:bin:/bin:/sbin/nologin
 3 daemon:x:2:2:daemon:/sbin:/sbin/nologin
 4 adm:x:3:4:adm:/var/adm:/sbin/nologin
 5 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
 6 sync:x:5:0:sync:/sbin:/bin/sync
 7 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
 8 halt:x:7:0:halt:/sbin:/sbin/halt
 9 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
10 operator:x:11:0:operator:/root:/sbin/nologin
11 games:x:12:100:games:/usr/games:/sbin/nologin
12 ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
13 nobody:x:99:99:Nobody:/:/sbin/nologin
14 systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
15 dbus:x:81:81:System message bus:/:/sbin/nologin
16 polkitd:x:999:998:User for polkitd:/:/sbin/nologin
17 libstoragemgmt:x:998:996:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
18 colord:x:997:995:User for colord:/var/lib/colord:/sbin/nologin
19 rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin
20 saslauth:x:996:76:Saslauthd user:/run/saslauthd:/sbin/nologin
21 abrt:x:173:173::/etc/abrt:/sbin/nologin
22 setroubleshoot:x:995:992::/var/lib/setroubleshoot:/sbin/nologin
23 rtkit:x:172:172:RealtimeKit:/proc:/sbin/nologin
24 gluster:x:994:991:GlusterFS daemons:/run/gluster:/sbin/nologin
25 chrony:x:993:990::/var/lib/chrony:/sbin/nologin
26 radvd:x:75:75:radvd user:/:/sbin/nologin
27 qemu:x:107:107:qemu user:/:/sbin/nologin
28 unbound:x:992:989:Unbound DNS resolver:/etc/unbound:/sbin/nologin
29 rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin
30 nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin
31 tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
32 usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin
33 geoclue:x:991:987:User for geoclue:/var/lib/geoclue:/sbin/nologin
34 pulse:x:171:171:PulseAudio System Daemon:/var/run/pulse:/sbin/nologin
35 saned:x:990:984:SANE scanner daemon user:/usr/share/sane:/sbin/nologin
36 gdm:x:42:42::/var/lib/gdm:/sbin/nologin
37 gnome-initial-setup:x:989:983::/run/gnome-initial-setup/:/sbin/nologin
38 sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
39 avahi:x:70:70:Avahi mDNS/DNS-SD Stack:/var/run/avahi-daemon:/sbin/nologin
40 postfix:x:89:89::/var/spool/postfix:/sbin/nologin
41 ntp:x:38:38::/etc/ntp:/sbin/nologin
42 tcpdump:x:72:72::/:/sbin/nologin

 4、grep -ni 忽略大小写显示包含root的行，且显示行号

1 [23:10:27 root@Centos8 ~]#grep -ni ROOT /etc/passwd
2 1:root:x:0:0:root:/root:/bin/bash
3 10:operator:x:11:0:operator:/root:/sbin/nologin

 5、grep  -c 统计匹配的行数

[23:26:39 root@Centos8 ~]#grep -c root /etc/passwd
2

 6、^表示行首，表示etc/passwd文件中root开头的行

[23:30:29 root@Centos8 ~]#grep ^root /etc/passwd
root:x:0:0:root:/root:/bin/bash

 

 二、SHELL脚本编程

 1、系统信息脚本显示系统：主机名、ip地址、版本号、内核版本号、CPU型号、内存、硬盘大小

 1 #!/bin/bash
 2 #
 3 #********************************************************************
 4 #Author:        gaojinzhou
 5 #QQ:            525184587
 6 #Date:          2020-04-02
 7 #FileName：     system_info.sh
 8 #URL:           https://www.cnblogs.com/gaojinzhou/
 9 #Description：      The test script
10 #Copyright (C):     2020 All rights reserved
11 #********************************************************************
12 RED="\E[1;31m"
13 GREEN="echo -e \E[1;32m"
14 END="\E[0m"
15 $GREEN-----------------------------------Host systeminfo----------------------------------------$END
16 echo -e "HOSTNAME:                    $RED`hostname`$END"                                        
17 echo -e "IPADDR:                      $RED`ifconfig ens160|grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | head -n1`$END"
18 echo -e "OSVERSION:                   $RED`cat /etc/redhat-release`$END"
19 echo -e "KERNEL:                      $RED`uname -r`$END"
20 echo -e "CPU:                         $RED`lscpu|grep 'Model name'|tr -s ' '|cut -d : -f2`$END"
21 echo -e "MEMORY:                      $RED`free -h|grep Mem|tr -s ' ' : |cut -d : -f2`$END"
22 echo -e "DISK:                        $RED`lsblk | grep '^sd'|tr -s ' ' |cut -d " " -f4`$END"
23 $GREEN------------------------------------------------------------------------------------------$END

输出结果如下：

 1 [23:46:20 root@Centos8 scripts]#. system_info.sh 
 2 -----------------------------------Host systeminfo----------------------------------------
 3 HOSTNAME:                    Centos8
 4 IPADDR:                      10.0.0.7
 5 OSVERSION:                   CentOS Linux release 8.1.1911 (Core) 
 6 KERNEL:                      4.18.0-147.el8.x86_64
 7 CPU:                          Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz
 8 MEMORY:                      1.9Gi
 9 DISK:                        200G
10 ------------------------------------------------------------------------------------------

 2、硬盘及磁盘节点编号自动告警脚本，将告警信息发送到邮箱，这里设置阈值为10

 1 #!/bin/bash
 2 #
 3 #********************************************************************
 4 #Author:        gaojinzhou
 5 #QQ:            525184587
 6 #Date:          2020-04-05
 7 #FileName：     disk_check.sh
 8 #URL:           https://www.cnblogs.com/gaojinzhou/
 9 #Description：      The test script
10 #Copyright (C):     2020 All rights reserved
11 #********************************************************************
12 WARNING=10
13 SPACE_USED=`df |grep '^/dev/sd'|grep -oE '[0-9]+%'|tr -d %| sort -nr|head -1`
14 INODE_USED=`df -i |grep '^/dev/sd'|grep -oE '[0-9]+%'|tr -d %| sort -nr|head -1`
15 [ "$SPACE_USED" -gt $WARNING -o "$INODE_USED" -gt $WARNING ] && echo "DISK USED:$SPACE_USED%,INODE_USED:$INODE_USED, will be full" | mail -s "DISK Warning" 525184587@qq.com    

 3、安全删除脚本，rm命令定义为mv命令实现安全删除

 1 #!/bin/bash
 2 #
 3 #********************************************************************
 4 #Author:        gaojinzhou
 5 #QQ:            525184587
 6 #Date:          2020-04-02
 7 #FileName：     rm.sh
 8 #URL:           https://www.cnblogs.com/gaojinzhou/
 9 #Description：      The test script
10 #Copyright (C):     2020 All rights reserved
11 #********************************************************************
12 WARNING_COLOR="echo -e \E[1;31m"
13 END="\E[0m"
14 DIR=/tmp/`date +%F_%H-%M-%S`
15 mkdir $DIR
16 mv  $*  $DIR
17 ${WARNING_COLOR} Move $* to $DIR $END 

 4、编写脚本 createuser.sh，实现如下功能：使用一个用户名做为参数，如果指定参数的用户存在，就显示其存在，否则添加之。并设置初始密码为123456，显示添加的用户的id号等信息，在第一次就提示用户立即改密码，如果没有参数，就提示用户“请输入用户名”

 1 #!/bin/bash
 2 #
 3 #********************************************************************
 4 #Author:        gaojinzhou
 5 #QQ:            525184587
 6 #Date:          2020-04-06
 7 #FileName：     createuser.sh
 8 #URL:           https://www.cnblogs.com/gaojinzhou/
 9 #Description：      The test script
10 #Copyright (C):     2020 All rights reserved
11 #********************************************************************
12 read -p "请输入用户名：" NAME
13 if getent passwd | grep $NAME; then
14     echo '该用户已存在'
15 else
16     useradd $NAME &> /dev/null
17     echo 123456 | passwd --stdin $NAME &> /dev/null
18     passwd -e $NAME &> /dev/null
19     echo -e  "$NAME 用户已创建成功，用户密码为123456，请立即更改密码"                                                                                                                        
20 fi

 

 5、编写脚本 yesorno.sh，提示用户输入yes或no,并判断用户输入的是yes还是no,或是其它信息

 

#!/bin/bash
#
#********************************************************************
#Author:        gaojinzhou
#QQ:            525184587
#Date:          2020-04-06
#FileName：     yesorno.sh
#URL:           https://www.cnblogs.com/gaojinzhou/
#Description：      The test script
#Copyright (C):     2020 All rights reserved
#********************************************************************
read -p "请输入yes或no：" INPUT
case $INPUT in
[Yy]|[Yy][Ee][Ss])
    echo "您输入的是yes"
;;
[Nn]|[Nn][Oo])
    echo "您输入的是no"
;;
*)
    echo "请输入yes或no"
esac