Pipe command shell filter

1. Use the command pipeline

Two or more commands (programs or processes) are connected together, the output of a command as the next command, two or more commands in this manner to form a connection duct (pipe) . Pipe using a pipe |connected to a plurality of commands, which is referred to pipe character.

Specific syntax Linux pipes are as follows:

command1 | command2
command1 | command2 [| commandN ...]

When the two commands is provided between the pipe, pipe character |command on the left to the right output becomes the input command. As long as the first command is written to standard output, and the second command is read from the standard input, then these two commands form a duct.

It should be noted, command1 must have the correct output, and command2 must be able to handle the output of command1; and command2 only properly handle the output of command1 , can not handle command1 error message.

2. Use the Pipe command of benefits:

command	effect
3 command to back up the database mysqldump -u root -p '123456' wiki > /tmp/wikidb.backup gzip -9 /tmp/wikidb.backup scp /tmp/wikidb.backup username@remote_ip:/backup/mysql/	mysqldump command is used to back up the database to a file wike /tmp/wikidb.backup. gzip command is used to compress large database files to save disk space; which -9represents the slowest speed compression best compression. scp command to copy the database backup files to the IP address of remote_ip backup server / backup / mysql / directory. Which usernameis remote server login user name, a password is required after the command is executed.
Use the pipe command mysqldump -u root -p '123456' wiki | gzip -9 | ssh username@remote_ip "cat > /backup/wikidb.gz"	The syntax of the command is compact and easy to use. By using a pipe, connected in series to the three commands together to complete complex tasks remotely mysql backup. Standard error from the output conduit to be mixed together. ssh username @ remote_ip "cat> /backup/wikidb.gz" is the command generated by the contents of the file get the file and save it to the cat command in /backup/wikidb.gz

3. The difference between the redirection and pipes

At first glance, there are pipes to redirect the role, it also changes the direction of the data input and output, then, in the end what is the difference between the pipe and redirect it?

Briefly, the redirection operator> Connect command file together with the file receiving command output; the pipe character | connect command command, a second command to receive the output of the first command . As follows:

command > file
command1 | command1

4.Linux pipe instance

		Export
The output of the ls command is sent to the grep command, view the file log.txt exists in the current directory	ls | grep log.txt	log.txt
Redirect the output conduit to a file	ls -al | grep log.txt >output.txt cat output.txt	-rw-r---RW. Mozhiyan mozhiyan 1 0 4 月 15 17:26 log.txt
Display information according to the user currently logged on the system after the user name sort	who | sort	mozhiyan: 0 2019-04-16 12:55 (: 0) mozhiyan pts / 0 2019-04-16 13:16 (: 0) Who command output as an input to the sort command, the command will show the two rear connection duct sorted by user name logged information of the user.

The pipeline and redirect

1) and the input redirection conduit

Input redirection operator <can be used in the pipeline to be used to obtain the input from a file, similar to the following syntax:

command1 < input.txt | command2
command1 < input.txt | command2 -option | command3

Features	command	Export
Use tr command to obtain input from os.txt file, and then send the output to the sort command or the like through the conduit uniq	cat os.txt ddd bbb CCC ccc tr a-z A-Z <os.txt | sort	BBB CCC CCC DDD

2) pipes and output redirection

You can also use the redirection operator> or >> standard output of the last command to redirect the pipeline, whose syntax is as follows:

command1 | command2 | ... | commandN > output.txt
command1 < input.txt | command2 | ... | commandN > output.txt

Features	command	Export
Use tr command os.txt content file is converted to upper case, and sorted using the sort command content, using deduplication uniq command line, and finally to redirect output to file ox.txt.new.	cat os.txt ddd bbb CCC ccc tr a-z A-Z <os.txt | sort|uniq>os.txt.new cat os.txt.new	BBB CCC DDD

6. Filter

Filter can be summarized in two points:

• 如果一个 Linux 命令是从标准输入接收它的输入数据，并在标准输出上产生它的输出数据（结果），那么这个命令就被称为过滤器。
• 过滤器通常与 Linux 管道一起使用。

常用的被作为过滤器使用的命令如下所示：

命令	说明
awk	用于文本处理的解释性程序设计语言，通常被作为数据提取和报告的工具。
cut	用于将每个输入文件（如果没有指定文件则为标准输入）的每行的指定部分输出到标准输出。
grep	用于搜索一个或多个文件中匹配指定模式的行。
tar	用于归档文件的应用程序。
head	用于读取文件的开头部分（默认是 10 行）。如果没有指定文件，则从标准输入读取。
paste	用于合并文件的行。
sed	用于过滤和转换文本的流编辑器。
sort	用于对文本文件的行进行排序。
split	用于将文件分割成块。
strings	用于打印文件中可打印的字符串。
tac	与 cat 命令的功能相反，用于倒序地显示文件或连接文件。
tail	用于显示文件的结尾部分。
tee	用于从标准输入读取内容并写入到标准输出和文件。
tr	用于转换或删除字符。
uniq	用于报告或忽略重复的行。
wc	用于打印文件中的总行数、单词数或字节数。

7.过滤器举栗

	功能	命令	输出	知识点
awk与管道结合	查看系统中的所有的账号名称，并按名称的字母顺序排序	awk -F: '{print $1}' /etc/passwd | sort|head -5	_apt asample backup bin daemon	-F： 以：作为分隔符，默认分隔符是空格 '{print $1}'打印第一列信息
	列出当前账号最常使用的 5个命令	history | awk '{print $2}' | sort | uniq -c | sort -rn | head -5	17 cat      14 vim      11 sudo       7 ll       7 cd	uniq -c 在每列旁边显示该行重复出现的次数。 sort -rn 按照数值的大小反向排序
	显示当前系统的总内存大小，单位为 KB	free | grep Mem | awk '{print $2}'	2029860	free命令可以显示Linux系统中空闲的、已用的物理内存及swap内存,及被内核使用的buffer
cut 与管道命令结合	查看系统中登录 Shell 是/bin/bash的用户名和对应的用户主目录的信息	grep "bin/bash" /etc/passwd|cut -d : -f1,6	root:/root roaddb:/home/roaddb asample:/home/asample	/ctc/passwd 文件用来存放用户账号的信息，文件中的每一行会记录一个账号的信息，每个字段之间用冒号分隔，第一个字段即是账号的账户名，第六个字段就是账号的主目录的路径。
	查看当前机器的CPU类型	cat /proc/cpuinfo |grep name|cut -d : -f 2|uniq	Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
	查看当前目录下的子目录数	ls -l|cut -c 1|grep d|wc -l	4	命令cut -c 1是截取每行的第一个字符
grep命令	查看系统日志文件中的错误信息	grep -i "error:" /var/log/messages | less		grep -iv [指定字条串] [文件] 在文件中搜索字符串匹配的行并输出 -i 不区分大小写 -v 排除指定字符串
	查看系统中 HTTP 服务的进程信息	ps aux | grep httpd
	查找我们的程序列表中所有命令名中包含关键字 zip 的命令	ls /bin /usr/bin | sort | uniq | grep zip	bunzip2 bzip2 bzip2recover funzip gpg-zip gunzip
	查找 /etc 目录下所有包含 IP 地址的文件	find /etc -type f -exec grep '[0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*[.][0-9][0-9]*' {} \;
tar命令	复制一个目录的整体结构	tar cf - /home/mozhiyan | ( cd /backup/; tar xf - )
	跨网络地复制一个目录的整体结构	tar cf - /home/mozhiyan | ssh remote_host "( cd /backup/; tar xf - )"
	跨网络地压缩复制一个目录的整体结构	tar czf - /home/mozhiyan | ssh remote_host "( cd /backup/; tar xzf - )"
	检査 tar 归档文件的大小，单位为字节	cd /; tar cf - etc | wc -c
	检查 tar 归档文件压缩为 tar.gz 归裆文件后所占的大小	tar czf - etc.tar | wc -c
	检查 tar 归档文件压缩为 tar.bz2 归裆文件后所占的大小	tar cjf - etc.tar | wc -c
head 命令	显示 ls 命令的前 10 行输出	ls /usr/bin | head	2to3 2to3-2.7 2to3-3.5 X11 [ aclocal aclocal-1.15 acyclic add-patch addpart	默认的输出行数为 10 行
	显示 ls 命令的前 5 行内容。	ls /usr/bin | head -n 5	2to3 2to3-2.7 2to3-3.5 X11 [
uniq命令	去掉输出中重复的行	sort testfile | uniq
	显示输出中各重复的行出现的次数，并按次数多少倒序显示	sort testfile | uniq -c | sort -nr
wc命令	统计当前登录到系统的用户数	who | wc -l	2	wc 命令用于统计包含在文本流中的字符数、单同数和行数
	统计当前的 Linux 系统中的进程数	ps -ef | wc -l	70