linux network analysis, performance analysis, text formatting, file read and write operations of the tool

Good tools will allow us to work more efficiently, the binding of work today under the linux share a few more useful tools.

Network Analysis Tool

mtr

mtr network link detection judgment is very useful tool, integrates the functions of these two tracert and ping commands, dynamic output a detection result. mtr default send icmp packets link detection, correlation node on the link will continue to do probing and the corresponding statistics, mtr can avoid the impact of fluctuations on the results of the node
where the middle line loss is not serious but the target address packet loss, probably because some of the icmp host routing protocol not treated or only allocate resources to deal with fixed limit, it is normal. Because icmp protocol request cpu resource consumption, in order to save costs, only the cpu fixed allocation request resource processing icmp

root@master1:~# mtr 114.114.114.114

The output states
of the first column (Host) : ip nodes and domain name
in the second column (Loss%) : node packet loss rate, packet loss is usually the final goal is the real loss
in the third column (Snt) : transmitting the packet the number of
the fourth column (last) : the last time the probe delay value of the
fifth column (Avg) : probe delay the average of
the sixth column (Best) : probe delay minimum, that the optimal value of
the seventh column (Wrst) : detecting the maximum value of delay, i.e., the worst value of
the eighth column (the StDev) : standard deviation

Optional Parameter Description
-h (- Help) : Provides help information
-v (- Version) : Displays the version
-c (- COUNT) : Set the number of ping limit value, this value is reached after the program exits
-r (- -report) : report output mode
-p (- Split) : each time tracking results are listed
-s (- PSize) : Specifies the size of the ping packet
-n (- DNS-NO) : not ip address do DNS
-a (- address) : ip address setting transmitted packet, when the host has a plurality ip
-i (- interval the) : set the interval between the return icmp, default lS
-4 : using ipv4 protocol
-6 : use the ipv6 protocol

Run-time interactive options
? Or H : Display help menu
d : switch the display mode
the n- : Enable or disable dns DNS
U : switch between icpm or udp packet detection

eg:
the ping number is 20, the packet size of 1024 bytes, the report output mode, the checking connectivity Baidu

Performance analysis tools

top

通过top命令查看系统的CPU、内存、运行时间、交换分区、执行的线程等信息。可以有效的发现系统的缺陷出在哪里，是内存不够、CPU处理能力不够还是IO读写问题等。
root@master1:~# top

输出结果说明
第一行：系统当前时间、系统运行时间、当前登录用户数、系统负载（1分钟、5分钟、15分钟的平均值）
第二行：tasks为任务进程，total进程总数，ruuning正在运行的进程数，sleeping睡眠的进程数，stopped停止运行的进程数，zombie僵尸进程数
第三行：cpu使用情况，us(user)用户空间占用cpu时间百分比，sy(system)内核空间占用cpu时间百分比，ni(nice)运行低优先级进程的cpu时间百分比，id(idle)空闲cpu时间百分比，wa(iowait)等待io占用cpu时间百分比，hi(hard interrupt)硬件硬中断占用cpu百分比，si(soft interrupt)软件软中断占用cpu百分比，st(steal)当前系统运行在虚拟机中的时候，被其他虚拟机占用的cpu时间百分比
整体的cpu使用率=1-id。当us很高时，证明cpu时间主要消耗在用户代码，需要优化用户代码。sy很高时，说明cpu时间都消耗在内核，要么是频繁的系统调用，要么是频繁的cpu切换（进程切换/线程切换）。wa很高时，说明有进程在进程频繁的IO操作，有可能是磁盘IO，也有可能是网络IO。si很高时，说明cpu时间消耗在处理软中断，网络收发包会触发系统软中断，所以大量的网络小包会导致软中断的频繁触发，典型的SYN Floor会导致si很高。
第四行：内存使用情况，total物理内存总大小，free空闲内存总量，used已经使用的内存量，buff表示用于读写磁盘缓存的内存，cache表示用于读写文件缓存的内存。avail表示可用的内存大小
第五行：虚拟内存信息， total表示能用的swap总量，swap free表示剩余，used表示已经使用的。
swap原理是把一块磁盘空间或者一个本地文件当成内存来使用，称为交换分区
第六行：具体的每个进程状态，PID进程id，USER进程所有者的用户名，PR进程调度优先级，NI进程nice值（优先级），越小的值代表越高的优先级，VIRT进程使用的虚拟内存，RES进程使用的物理内存（不包括共享内存），SHR 进程使用的共享内存大小，S进程状态(D：不可中断的睡眠状态，R：运行，S：睡眠，T：跟踪/停止，Z：僵尸进程)，%CPU 进程使用的cpu占用百分比，%MEM 进程使用的内存占用百分比，TIME+ 进程启动后到现在所用的全部cpu时间，COMMAND 进程的启动命令（默认只显示二进制，-c参数能够显示命令行和启动参数）

可选参数说明
-b 批处理模式操作
-d 指定每两次屏幕信息刷新之间的时间间隔
-p 通过指定监控进程ID来仅仅监控某个进程的状态
-q 使top没有任何延迟的进行刷新，如果调用程序有超级用户权限，那么top将以尽可能高的优先级运行
-S 指定累积模式
-s 使top命令在安全模式中运行，这将去除交互命令所带来的潜在危险。
-i 使top不显示任何闲置或者僵死进程
-c 显示整个命令行而不只是显示命令名

运行时交互式选项
h或者? 显示快捷键帮助
q 退出程序
1 展开多核cpu显示
m 切换显示内存信息
M 根据内存使用大小排序
P 根据CPU使用率进行排序（默认排序）
c 切换显示命令名称和完整命令行
k 终止一个进程
i 忽略闲置和僵死进程
r 重新安排一个进程的优先级别，系统提示用户输入需要改变的进程PID以及需要设置的进程优先级
S 切换到累计模式
s 更改刷新间隔时间，单位秒，默认是5s
f或者F 从当前显示中添加或者删除项目
o或者O 改变显示项目的顺序
l 切换显示平均负载和启动时间信息
t 切换显示进程和CPU状态信息
T 根据时间/累计时间进行排序
w 将当前设置写入~/.toprc文件中

格式化工具

jq

jq可以对json数据进行分片、过滤、映射和转换，可以让linux命令和shell脚本在处理json数据时变得得心应手。
默认linux系统是不带jq命令的，首先需要安装

root@master1:~# apt-get install -y jq

有如下json格式的文件

root@master1:~# cat test.txt
[{"address":{"province":"hubei","city":"wuhan"},"site":"www.hubei.com"},{"address":{"province":"hunan","city":"changsha"},"site":"www.hunan.com"},{"address":{"province":"zhejiang","city":"hangzhou"},"site":"www.zhejiang.com"}]

Now formatted output test.txt

Or formatted as follows

According to the index to see elements
such as viewing the first element

Note: The file must first be treated with jq is in line with json format, or with jq formatting error, jq original file formatting will not change the layout format

File manipulation tool

sponge

sponge is a relatively easy to use tools to modify the file, support for reading all input before the file is written, it is more useful when writing to the same file again after reading the file
such as the following file

root@master1:~# cat ceshi.txt 
windowns
linux
c language
python language
go language
hello world

Now we need the files in alphabetical order according to the first re-save the file to

root@master1:~# sort ceshi.txt 
c language
go language
hello world
linux
python language
windowns

You might think of as redirect

But you will find this to redirect the source file is empty, of course, this is not what we want to see.
Now we will re-write the contents of the file

root@master1:~# cat ceshi.txt
windowns
linux
c language
python language
go language
hello world

At this point you might think would redirect the temporary files by way

of course to meet our needs through temporary files

Because the sponge can read all input before the file is written, it can be a good solution to our needs by sponge
Linux default is no sponge command, you first need to install

root@master1:~# apt-get install -y moreutils

Introducing jq tools above, we formatted output test.txt, but test.txt file itself has not changed, so now we can jq and sponge used in combination formatted file test.txt

root@master1:~# cat test.txt
[{"address":{"province":"hubei","city":"wuhan"},"site":"www.hubei.com"},{"address":{"province":"hunan","city":"changsha"},"site":"www.hunan.com"},{"address":{"province":"zhejiang","city":"hangzhou"},"site":"www.zhejiang.com"}]