Linux basic study notes-file formatting

File formatting

1. Sed tool

[root@li ~]# sed [-nefr] [动作]
选项与参数：
-n：使用安静模式。在一般的 sed 用法中，所有来自 STDIN 的数据一般都会被列出到屏幕上。如果加上 -n 参数后，只有经过处理的那一	  行才会被列出来
-e：直接在指令列模式上进行 sed 的动作编辑
-f：直接将 sed 的动作写在一个文件内
-r：sed 的动作支持的是延伸的正则表达式
-i：直接修改读取文件内容，而不是屏幕输出

动作说明：	'[n1[,n2]]function'（一定要有单引号）
n1,n2：不见得存在，一般代表 “选择进行动作的行数”

function：
a：新增，a 的后面可以接字符串，而这些字符串会在新的一行出现（目前的下一行）
c：取代，c 的后面可以接字符串，这些字符串可以取代 n1，n2 之间的行
d：删除
i：插入，i 后面可以接字符串，而这些字符串会在新的一行出现（目前的上一行）
p：打印，通常 p 会将参数 sed -n 一起出现
s：取代，可以直接进行取代的工作。通常这个 s 的动作可以搭配正则表达式

1.1. Add/delete functions by line

#将 /etc/passwd 的内容列出来并打印行号，同时，删除第 2~5 行
[root@li ~]# nl /etc/passwd | sed '2,5d'
     1  root:x:0:0:root:/root:/bin/bash
     6  sync:x:5:0:sync:/sbin:/bin/sync
     7  shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
     8  halt:x:7:0:halt:/sbin:/sbin/halt
     9  mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
     ...
#在第二行后（也就是加在第三行）加上 drink tea 的字样
[root@li ~]# nl /etc/passwd | sed '2a drink tea'
     1  root:x:0:0:root:/root:/bin/bash
     2  bin:x:1:1:bin:/bin:/sbin/nologin
drink tea
	 ...
#在第二行后加上两行字（要使用反斜杠）
[root@li ~]# nl /etc/passwd | sed '2a drink tea ...\
> drink beer?'
     1  root:x:0:0:root:/root:/bin/bash
     2  bin:x:1:1:bin:/bin:/sbin/nologin
drink tea ...
drink beer?
...

1.2. Replacement and display function by behavior unit

#将第 2~5 行取代为 No 2-5 number
[root@li ~]# nl /etc/passwd | sed '2,5c No 2-5 number'
     1  root:x:0:0:root:/root:/bin/bash
No 2-5 number
     6  sync:x:5:0:sync:/sbin:/bin/sync
     ...
#仅列出第 5~7 行数据
[root@li ~]# nl /etc/passwd | sed -n '5,7p'
     5  lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
     6  sync:x:5:0:sync:/sbin:/bin/sync
     7  shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown

1.3. The function of searching and replacing some data

sed 's/要被取代的字符串/新的字符串/g'

#先用 ifconfig 查询 IP
[root@li ~]# ifconfig ens33
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.246  netmask 255.0.0.0  broadcast 10.255.255.255
        inet6 fe80::91cd:d416:48f5:585d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:5d:05:76  txqueuelen 1000  (Ethernet)
        RX packets 4950  bytes 333302 (325.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8784  bytes 1012348 (988.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
#利用关键字配合 grep 截取出关键的一行数据
[root@li ~]# ifconfig ens33 | grep 'inet '
        inet 10.0.0.246  netmask 255.0.0.0  broadcast 10.255.255.255
#将 IP 前面的部分删除        
[root@li ~]# ifconfig ens33 | grep 'inet ' | sed 's/^.*inet //g'
10.0.0.246  netmask 255.0.0.0  broadcast 10.255.255.255
#将 IP 后面的部分删除
[root@li ~]# ifconfig ens33 | grep 'inet ' | sed 's/^.*inet //g' | sed 's/ *netmask.*$//g'
10.0.0.246

#先用 grep 将关键字 MAN 所在行取出来
[root@li ~]# cat /etc/man_db.conf | grep 'MAN'
# MANDATORY_MANPATH                     manpath_element
# MANPATH_MAP           path_element    manpath_element
# MANDB_MAP             global_manpath  [relative_catpath]
# every automatically generated MANPATH includes these fields
...
#删掉批注后的数据
[root@li ~]# cat /etc/man_db.conf | grep 'MAN' | sed 's/#.*$//g'



MANDATORY_MANPATH                       /usr/man
MANDATORY_MANPATH                       /usr/share/man
MANDATORY_MANPATH                       /usr/local/share/man
...
#删掉空白行
[root@li ~]# cat /etc/man_db.conf | grep 'MAN' | sed 's/#.*$//g' | sed '/^$/d'
MANDATORY_MANPATH                       /usr/man
MANDATORY_MANPATH                       /usr/share/man
MANDATORY_MANPATH                       /usr/local/share/man

1.4, directly modify the content of the file

Very dangerous operation, related to parameter -i .

---

2. File formatting and related operations

2.1. Format printing: printf

[root@li ~]# printf '打印格式' 实际内容
选项与参数：
关于格式方面的几个特殊样式：
	\a		警告声音输出
	\b		退格键
	\f		清除屏幕
	\n		输出新的一行
	\r		即 Enter 键
	\t		水平的 tab 键
	\v		垂直的 tab 键
	\xNN	NN 为两位数字，可以转换数字为字符
关于 C 语言内，常见的变数格式：
	%ns		那个 n 是数字，s 代表 string，即多少字符
	%ni		那个 n 是数字，i 代表 int，即多少整数字数
	%N.nf	小数

2.2, awk: a useful data processing tool

[root@li ~]# awk '条件类型1{动作1} 条件类型2{动作2} ...' 文件名

[root@li ~]# last -n 5
root     pts/0        10.0.0.100       Fri Aug  7 08:26   still logged in
reboot   system boot  3.10.0-1127.el7. Fri Aug  7 08:24 - 09:04  (00:39)
root     pts/0        li-911m.lan      Thu Aug  6 15:40 - crash  (16:44)
reboot   system boot  3.10.0-1127.el7. Thu Aug  6 15:40 - 09:04  (17:23)
root     pts/0        li-911m.lan      Thu Aug  6 08:56 - crash  (06:43)
#取出账号与账号的 IP
[root@li ~]# last -n 5 | awk '{print $1 "\t" $3}'
root    10.0.0.100
reboot  boot
root    li-911m.lan
reboot  boot
root    li-911m.lan

The entire awk processing flow:

• Read in the first row, and fill in the data of the first row in $0, $1, $2,...;
• According to the restriction of "condition type", judge whether the following "action" is needed;
• Complete all actions and condition types;
• If there are subsequent "rows" of data, repeat the previous steps 1~3 until all the data is read.

Awk is "a unit of processing at one time", and "a field is the smallest unit of processing" . So how does awk know how many rows are in this data? This requires the help of awk's built-in variables:

Variable name	Description
NF	The total number of fields in each row ($0)
NO	What awk currently deals with is the meaning of "the few lines"
FS	The current separator character, the default is space

We continue to use last -n 5 as an example, if we want:

• List the account number of each line (that is, $1);
• List the number of rows currently processed (the NR variable in awk);
• And explain how many fields the line has (that is, the NF variable in awk).

[root@li ~]# last -n 5 | awk '{print $1 "\t lines:" NR "\t columns: " NF}'
root     lines:1         columns: 10
reboot   lines:2         columns: 11
root     lines:3         columns: 10
reboot   lines:4         columns: 11
root     lines:5         columns: 10

Logical operators of awk:

Operation unit	Description
>	more than the
<	Less than
>=	greater or equal to
<=	Less than or equal to
==	equal
!=	not equal to

#第三栏小于 10 的数据，仅列出账号与第三栏
[root@li ~]# cat /etc/passwd | awk '{FS=":"} $3 < 10 {print $1 "\t " $3}'
root:x:0:0:root:/root:/bin/bash
bin      1
daemon   2
...

Why is the first line not displayed correctly? Because when we read the first line, those variables $1, $2,... are still separated by spaces by default, so the defined FS=":" only takes effect on the second line. Solution: Use the BEGIN keyword:

[root@li ~]# cat /etc/passwd | awk 'BEGIN {FS=":"} $3 < 10 {print $1 "\t " $3}'
root     0
bin      1
daemon   2

2.3. File comparison tool

2.3.1、diff

diff is the ratio of the two documents difference, and is in units to alignment! Generally used in the comparison of ASCII plain text files. Since the comparison is performed in units of lines, diff is usually used for the difference between the new and old versions of the same file (or software) .

[root@li ~]# diff [-bBi] from-file to-file
选项与参数：
from-file：原始比对文件
to-file：目的比对文件
from-file 或 to-file 可以用 - 取代

-b：忽略一行当中，仅有多个空白的差异
-B：忽略空白行的差异
-i：忽略大小写的差异

[root@li ~]# mkdir -p /tmp/testpw
[root@li ~]# cd /tmp/testpw/
[root@li testpw]# cp /etc/passwd passwd.old
[root@li testpw]# cat /etc/passwd | sed -e '4d' -e '6c no six line' > passwd.new
[root@li testpw]# diff passwd.old passwd.new
4d3					#左边第四行被删除了，基准是右边的第三行
< adm:x:3:4:adm:/var/adm:/sbin/nologin		#被删除的那一行
6c5					#左边文件第六行被取代了
< sync:x:5:0:sync:/sbin:/bin/sync			#被取代的内容
---
> no six line		#右边文件的第五行

[root@li testpw]# diff /etc/rc0.d/ /etc/rc5.d/
只在 /etc/rc0.d/ 存在：K90network
只在 /etc/rc5.d/ 存在：S10network

2.3.2、cmp

cmp is mainly compared in bytes .

[root@li ~]# cmp [-l] 文件1 文件2
选项与参数：
-l：将所有不同点的字节列出来。因为 cmp 默认仅列出第一个不同点

[root@li testpw]# cmp passwd.old passwd.new
passwd.old passwd.new 不同：第 106 字节，第 4 行		#默认仅列出第一个不同点
[root@li testpw]# cmp -l passwd.old passwd.new
106 141 154
107 144 160
108 155  72
...
#内容太多

2.3.3、patch

Patch and diff are inseparable. diff is used to compare the differences between two files. How to upgrade? That is, " first compare the differences between the new and the old version, and make the difference document into a patch, and then update the old file from the patch ."

[root@li testpw]# diff -Naur passwd.old passwd.new > passwd.patch
[root@li testpw]# cat passwd.patch
--- passwd.old  2020-08-07 09:29:25.309779367 +0800
+++ passwd.new  2020-08-07 09:30:05.133777094 +0800
@@ -1,9 +1,8 @@
 root:x:0:0:root:/root:/bin/bash
 bin:x:1:1:bin:/bin:/sbin/nologin
 daemon:x:2:2:daemon:/sbin:/sbin/nologin
-adm:x:3:4:adm:/var/adm:/sbin/nologin
 lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
-sync:x:5:0:sync:/sbin:/bin/sync
+no six line
 shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
 halt:x:7:0:halt:/sbin:/sbin/halt
 mail:x:8:12:mail:/var/spool/mail:/sbin/nologin

[root@li testpw]# patch -pN < patch_file		#更新
[root@li testpw]# patch -R -pN < patch_file		#还原

#更新文件
[root@li testpw]# patch -p0 < passwd.patch
patching file passwd.old
[root@li testpw]# ll passwd*
-rw-r--r--. 1 root root 778 8月   7 09:30 passwd.new
-rw-r--r--. 1 root root 778 8月   7 09:43 passwd.old		#文件完全一样了
#还原文件
[root@li testpw]# patch -R -p0 < passwd.patch
patching file passwd.old
[root@li testpw]# ll passwd*
-rw-r--r--. 1 root root 778 8月   7 09:30 passwd.new
-rw-r--r--. 1 root root 835 8月   7 09:44 passwd.old		#不一样了

2.4. File printing preparation: pr

[root@li testpw]# pr /etc/man_db.conf


2018-10-31 04:26                /etc/man_db.conf                 第 1 页


#
#
# This file is used by the man-db package to configure the man and cat paths.
...