After the bash command is executed, usually the command will return a result. How to do some operations on the returned result? Then you have to use the pipe command '|'.
The above paragraph briefly talks about the role of the pipeline command, so what is the pipeline command?
Answer: A very classic sentence, remember, what is a pipeline, a pipeline is to pass the result returned by the previous command to the command behind the pipeline for processing. It is precisely because the command after the pipeline is used to process the return result of the command before the pipeline, so the pipeline command must be able to accept standard input, so it can be called a pipeline command.
Let's take a look at an example first. Suppose we want to know how many files are under /etc/, then we can use ls /etc to view them. However, because there are too many files under /etc, the screen is filled up in one breath. Knowing what the content of the previous output is, at this time we can cooperate with less bright, as follows
[root@bogon shell]# ls -al /etc | less total 1816 drwxr-xr-x. 100 root root 4096 Aug 17 19:09 . dr-xr-xr-x. 21 root root 4096 Aug 17 18:31 .. drwxr-xr-x. 3 root root 4096 Apr 25 08:59 abrt
According to my definition of pipeline above, it can be seen that the result returned by ls -al /etc has been processed by less, so it is not so simple, but it is worth noting that the command "|" (pipe) can only Processing, the previous command is executed correctly and returns the result without error. If an error occurs in the previous command of the pipeline, it cannot be processed.
Common Pipeline Commands
1. Select the command: cut, grep
What is the choice command? To put it bluntly, it is to extract what we want after analyzing a piece of data. However, it should be noted that, in general, the selection information is usually analyzed for the "line", not the entire information analysis.
[ cat ]: This command can "cut" a certain segment of a piece of information, and the processed information is in "line" units.
grammar: cat [-dfc] parameter: -d: followed by a delimiter, used with -f; -f: Divide a piece of information into several paragraphs according to the separator of -d, and use -f to take out the meaning of the first paragraph -c: Take out the fixed character range in units of characters Example 1: Take out the PATH and take out the fifth segment [root@bogon ~]# echo $PATH /usr/java/jdk1.7.0_55/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin [root@bogon ~]# echo $PATH | cut -d ':' -f 5 /bin What if you want to list paragraphs 3 and 5? As follows [root@bogon ~]# echo $PATH | cut -d ':' -f 3,5 /usr/local/bin:/bin [root@bogon ~]# Example 2: Obtain all characters after the 12th character from the information output by export [root@bogon ~]# export declare -x CLASSPATH=".:/usr/java/jdk1.7.0_55/lib/dt.jar:/usr/java/jdk1.7.0_55/lib/tools.jar" declare -x G_BROKEN_FILENAMES="1" declare -x HISTCONTROL="ignoredups" .....! [root@bogon ~]# export | cut -c 12- CLASSPATH=".:/usr/java/jdk1.7.0_55/lib/dt.jar:/usr/java/jdk1.7.0_55/lib/tools.jar" G_BROKEN_FILENAMES="1" HISTCONTROL="ignoredups" HISTSIZE="1000" ....! # Note that each data is output neatly. If we want to "declare -x", do this Example 3: Use last to leave only the first name in the information that displays the login [root@bogon ~]# last root pts/0 192.168.238.1 Sun Aug 17 19:52 still logged in root pts/0 192.168.238.1 Sun Aug 17 18:54 - 19:25 (00:30) reboot system boot 2.6.32-358.el6.i Sun Aug 17 18:31 - 20:02 (01:31) .....! [root@bogon ~]# last | cut -d ' ' -f1 root root reboot .....
Remarks: The main function of cut is to decompose the data in the same line, aiming at the line, but cut is very laborious when dealing with data connected with multiple spaces.
[ grep ]: The cut command just now takes out a certain part of the information in the same line of information, while grep analyzes a line of information. If there is information we need, take out the line.
grammar: grep [-acinv] [--color=auto] 'Search string' fileName parameter: -a: Find data in the way of binary file and text file -c: count the number of 'find strings' found -i: ignore the difference in case, so the case is considered the same; (important) -n: output line number by the way; (important) -v: reverse selection, i.e. show those lines where the 'lookup string' was not found (important) --color=auto: can add color to the found keyword part (it works well in combination with the print tool) Example 1: When there is a line with root in the last, take it out [root@bogon ~]# last | grep root root pts/0 192.168.238.1 Sun Aug 17 19:52 still logged in root pts/0 192.168.238.1 Sun Aug 17 18:54 - 19:25 (00:30) root tty1 Fri Aug 15 03:00 - down (00:00) ...... Example 2: When there is no root line in the last, take it out [root@bogon ~]# last | grep -v root reboot system boot 2.6.32-358.el6.i Sun Aug 17 18:31 - 20:15 (01:43) reboot system boot 2.6.32-358.el6.i Thu Aug 14 18:16 - 03:00 (08:43) reboot system boot 2.6.32-358.el6.i Wed Aug 13 22:37 - 03:01 (04:24) ...... Example 3: In the output information of last, as long as there is root, it is taken out, and only the first column is taken [root@bogon ~]# last | grep 'root' | cut -d ' ' -f1 root root root ...... Example 4: Bold the detected data [root@bogon ~]# cat bb.txt | grep --color=auto 'name' my name is ickes
2. Sort commands: sort, wc, uniq
Many times, we will calculate the total number of data of the same type in the data. For example, we can use last to find out which users have logged in to the host this month. So can we find out, for each user, the total number of times they have logged in? At this point, commands such as sorting and calculation are needed to assist.
【sort】 : Sort command, which can be sorted according to different data types.
grammar: sort [-fbMnrtuk] [file or stdin (standard input)] parameter: -f: ignore case differences, sort -b: Sort by ignoring the leading space part; -M: sort with the name of the month, for example: sort l of JAN, DEC, etc. -n: Use pure numbers to sort (default is to sort by character type) -r: reverse sort -u: means uniq, in the same data, only one row is selected as the representative -t: separator, the default is to use the [Tab] key to separate -k: the meaning of sorting by that interval Example 1: All personal accounts are recorded under /etc/passwd, and the accounts are sorted #sort defaults to "first" data to sort, and it is sorted by character type, so it is the order of az [root@bogon ~]# cat /etc/passwd | sort abrt:x:173:173::/etc/abrt:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin Example 2: The content of /etc/passwd is divided by: ":", I want to sort by the third column [root@bogon ~]# cat /etc/passwd | sort -t ':' -k 3 root:x:0:0:root:/root:/bin/bash uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin usbmuxd:x:113:113:usbmuxd user:/:/sbin/nologin bin:x:1:1:bin:/bin:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin .... #You will find that the third column is a number, but it is unordered, why? #Answer: Originally this is the case, because the default is to sort by characters # Change it to the following and it's OK [root@bogon ~]# cat /etc/passwd | sort -t ':' -k 3 -n root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin ..... Example 3: Comprehensive example, use last to take only the account number and sort the output data [root@bogon ~]# last | cut -d ' ' -f1 | sort
[uniq] : If the sorting is completed, and you want to list only one display of the duplicate data (that is, deduplication), how to do it? Then you have to use the uniq command.
grammar:
uniq [-ic]
parameter:
-i: ignore the difference between upper and lower case characters;
-c: count
Example 1: Use last to list the accounts, take out only the account column, sort and then deduplicate
[root@bogon ~]# last | cut -d ' ' -f1 | sort | uniq
ickes
reboot
root
wtmp
Example 2: If on the basis of example 1, I also need to know the number of times each person has logged in (that is, the number of times of each occurrence after deduplication)
[root@bogon ~]# last | cut -d ' ' -f1 | sort | uniq -c
3 ickes
71 reboot
194 root
1 wtmp
The uniq command is so simple
[ wc ]: This command can help us count how many lines, words, and characters in a file
grammar:
wc [-lwm]
parameter:
-l: only list how many lines there are
-w: only list how many words there are
-m: only list how many characters there are
Example 1: List, how many words, lines and characters are in the file /etc/passwd
[root@bogon ~]# cat /etc/passwd | wc
31 45 1432
#The meanings of the three output numbers are: line, number of words, number of characters
3. Character conversion commands: tr, col, join, paste, expand
【tr】: used to delete the text in a piece of information, or to replace the text information
grammar: tr [-d] 'set1'... parameter: -d: delete the string set1 in the message Example 1: Delete the ":" in the file /etc/passwd [root@bogon ~]# cat /etc/passwd | tr -d ':' rootx00root/root/bin/bash binx11bin/bin/sbin/nologin daemonx22daemon/sbin/sbin/nologin .... Example 2: Convert the file /etc/passwd to uppercase [root@bogon ~]# cat /etc/passwd | tr '[a-z]' '[A-Z]' ROOT:X:0:0:ROOT:/ROOT:/BIN/BASH BIN:X:1:1:BIN:/BIN:/SBIN/NOLOGIN DAEMON:X:2:2:DAEMON:/SBIN:/SBIN/NOLOGIN .....
[ col ]: The only requirement I use for this command is to convert the tab key to the equivalent space bar to facilitate the processing of subsequent commands
The use example is as follows: Convert the tab key in the /etc/man.config file to the equivalent space bar
#Use cat -A to display all special keys [root@bogon ~]# cat -A /etc/man.config #At this point you find that there are a lot of ^I this is the tab key #Use col to convert tab to space bar [root@bogon ~]# cat /etc/man.config | col -x | cat -A
[ join ]: You can understand the meaning of join from the literal meaning. Yes, it is used to connect the data between two files, and it mainly loads the line with the same data in the two files together.
grammar:
join [-ti12] file1 file2
parameter:
-t:join splits the data by a space character by default, and compares the data of the "first field",
If the fields are equal, connect the two data into one line, and the first field is placed first;
-i: ignore differences in case
-1: This is the number 1, which means that the first file should be analyzed by that field
-2: Represents the meaning of the second file to be analyzed by that field
First look at my test data
[root@bogon bash]# cat aa.txt bb.txt
---------------aa.txt
aa aad
bb bbd
cc ccd
----------------bb.txt
aa aad1
bb bbd1
cc ccd1
Example 1: Integrate aa.txt bb.txt related data into one column
[root@bogon bash]# join -t ' ' ./aa.txt ./bb.txt
aa aad aad1
bb bbd bbd1
cc ccd ccd1
#Through this operation, we can integrate the same first field of the two files into one line
#The same field of the second file will not be displayed (because it is already the first line)
Example 2: We know that the fourth field of /etc/passwd is GID,
The GID is recorded in the third field of /etc/group. The following example is the integration
[root@bogon bash]# join -t ':' -1 4 /etc/passwd -2 3 /etc/group
0:root:x:0:root:/root:/bin/bash:root:x:
1:bin:x:1:bin:/bin:/sbin/nologin:bin:x:bin,daemon
......
With this command, it is important to note that before using join, the files you need to process should be sorted in advance. Otherwise, some comparison projects will be ignored, and I have been there, and there are always problems with the data.
[ past ]: This also means connecting two files, but it is much simpler than join. Compared with join, the data correlation of the two files must be compared. Past is to directly add two lines together and separate them with the tab key.
grammar: paste [-d] file1 file2 parameter: -d: can be followed by a separator character, the default is to split by the (tab) key Example: Paste /etc/passwd and /etc/shadow on the same line [root@bogon bash]# paste aa.txt bb.txt aa aad aa aad1 bb bbd bb bbd1 cc ccd cc ccd1 [root@bogon bash]# paste -d ':' aa.txt bb.txt aa aad: aa aad1 bb bbd: bb bbd1 cc ccd:cc ccd1
[ expand ]: This command is a bit interesting, it is to convert the tab key into a space bar, which is very different from the previous col command
Compare expand and col:
expand: converts the tab key to the equivalent 8 spacebars, and col converts the tab key to a spacebar
grammar: expand [-t] file -t: can be followed by numbers. In general, a tab key can be replaced by 8 spacebars. We can also customize how many characters a tab key represents. Example 1: Take out the line whose line head is MANPATH in /etc/man.config, and only take the first three lines [root@bogon bash]# grep '^MANPATH' /etc/man.config | head -n 3 MANPATH /usr/man MANPATH /usr/share/man MANPATH /usr/local/man You have new mail in /var/spool/mail/root #^ This symbol is the logo of the beginning of the line. I will analyze this when I review regular expressions. It is very interesting. Example 2: List all the symbols in the above example [root@bogon bash]# grep '^MANPATH' /etc/man.config | head -n 3 | cat -A MANPATH^I/usr/man$ MANPATH^I/usr/share/man$ MANPATH^I/usr/local/man$ #tab key is ^I Example 3: Replace the above tab key with 6 space bars [root@bogon bash]# grep '^MANPATH' /etc/man.config | head -n 3 | expand -t 6 | cat -A MANPATH /usr/man$ MANPATH /usr/share/man$ MANPATH /usr/local/man$ #Is there no ^I anymore
4. Cutting command: split
[split] This command can split a large file into small files according to the size of the file or the number of lines.
grammar: split [-bl] file PREFIX parameter: -b: The size of the cutting file can be followed, and the unit can be added, such as b, k, m, etc. -l: split by the number of lines; PREFIX: Represents the leading character, which can be used as the leading text of the cutting file Example 1: The install.log in my user directory has 44k, and it is divided into 10k files [root@bogon ~]# du -sh install.log 44K install.log [root@bogon ~]# split -b 10k ./install.log split [root@bogon ~]# ll -k split* -rw-r-r--. 1 root root 10 Aug 18 02:11 split -rw-r--r--. 1 root root 10 Aug 18 02:11 splitab -rw-r--r--. 1 root root 10 Aug 18 02:11 splitac -rw-r--r--. 1 root root 9 Aug 18 02:11 splitad #The last split is the leading character. At this time, the system will name it with xxxxaa, xxxxab, xxxxac... Example 2: If the above three small files are combined into a large file bak.log [root@bogon ~]# cat splita* >> bak.log #Is it so simple, just use data stream redirection Example 3: In the information output by ls -al /, every 3 lines are recorded as a file [root@bogon bash]# ls -al / | split -l 3 - s [root@bogon bash]# wc -l s* 3 saa 3 sab 3 sac 3 sad 3 leaves 3 pure 3 sag 2 saw 23 total #The focus is on that -. Generally speaking, if stdout (standard output)/stdin (standard input) is required, but there is no file, #Some are just -, then that - will be treated as stdout or stout
About - (the meaning of the minus sign, I will analyze it in the next article)