sed is the abbreviation of stream editor, which is called "stream editor" in Chinese.
The sed command is a line-oriented processing tool. It uses "line" as the processing unit and processes each line. The processed results will be output to the standard output (STDOUT). You will find that the sed command is a very polite command. It will not make any rash modifications to the files it reads, but will output the contents to the standard output.
Let's take a look at the command format of sed:
sed command file
- Command part: The processing to be performed on the content of each line (this part is very important).
- file part: the file to be processed. If the file parameter is omitted, sed will use standard input as the processing object.
How does sed work?
As we just said, the sed command is "line" oriented, processing one line of content at a time. During processing, sed will store the lines to be processed in the buffer, and then use the sed command to process the contents of the buffer. After the processing is completed, the contents of the buffer will be sent to the screen. Then process the next line, and repeat until the end of the file. This buffer is called "pattern space".
As mentioned earlier, during this process, the sed command does not make any changes to the file itself.
Let's take a look at the simplest example of the sed command to give everyone a perceptual understanding of it.
#Let's first take a look at the content of the original file [roc@roclinux ~]$ cat roc.txt test 1 test2 testtest XtestX BBtest #We want to use the sed command to delete the line with the character "2" in the file [roc@roclinux ~] $ sed '/2/d' roc.txt test 1 testtest XtestX BBtest
This example uses sed to delete lines containing the character "2" in the roc.txt file. You see, the example is very simple, the command part of this command is /2/d, and it is enclosed in single quotes. You must also learn to do this, use sed, and don't forget to enclose the command part in single quotes. /2/dThe d in means delete, which means that as long as the content of a certain line contains the character 2, this line will be deleted. (The so-called deletions by sed are all performed in the pattern space and will not actually change the original roc.txt file.)
Want to use sed command to achieve the effect of cut command
If we want to achieve an effect similar to cut-d:-f 1/etc/passwd, that is, to extract the first domain using colon as the separator, how should we use the sed command?
#Let’s take a look at the contents of the /etc/passwd file first [roc@roclinux ~]$ head -n 5 /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1 :1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp: x:4:7:lp:/var/spool/lpd:/sbin/nologin #This time we use the sed command to extract the first field of each line in the file. The separator is colon [roc@roclinux ~]$ head -n 5 /etc/passwd|sed 's/:.*$//' root bin daemon adm lp
As you can see, we have specified the command part 's/：.*$//', which means that we need to clear the part from the first colon to the end of each line, so that what is left is the content before the first colon.
What are the useful options of sed?
When it comes to the options of the sed command, we have to mention -nthe options. If we want to explain this option clearly, it still takes some brains and writing.
As mentioned earlier, sed will process the lines in the pattern space and output them to standard output. This is the default processing method. In other words, unless you use "d" to delete this line, all lines processed by the "pattern space" will be output to the standard output (screen). Let’s take a look at the following example:
#Let’s take a look at the content of the original file first [roc@roclinux ~]$ cat roc.txt 1 2 3 4 5 #Look carefully, there are two "4"s in the output [roc@roclinux ~]$ sed '/4 /p' roc.txt 1 2 3 4 4 5
Look, all the original file contents are output, and the line containing the character 4 is output twice.
But this is how the sed command works. It will output the processed lines first and then perform subsequent actions. (Here we set p, which means printing this line.) This obviously does not meet our original intention. We just want the sed command to find the line containing 4 and then output it.
At this time, you might as well add -nthe option and give it a try. You will find that the result becomes what you want.
[roc@roclinux ~]$ sed -n '/4/p' roc.txt 4
-nThe option will sternly warn the sed command: Don't give me random output unless it is a line that is clearly indicated to be output. -nThe option is often used in conjunction with p, which means to output those matching lines.
There are many tricks in the command part
Remember what we introduced earlier, the command format of the sed command is as follows:
$ sed command file
Among them, the command part is the essence of the sed command. Your mastery of the command part determines whether you are a sed master.
The command part can be divided into two pieces of knowledge: one is range setting and the other is action processing.
Scoping can be expressed in two different ways:
- Specify the number of lines: for example, '3,5' means lines 3, 4 and 5; and '5,$' means line 5 to the last line of the file.
- Pattern matching: For example, /^[^dD]/ means matching lines that do not start with d or D.
The action processing part will provide a wide range of actions for you to choose from. Here are some of the most commonly used actions:
- d: Indicates deleting rows.
- p: Print this line.
- r: Read the contents of the specified file.
- w: Write to the specified file.
- a: Insert a new line and new content below.
It's time to show off the power of sed
Facts speak louder than words. We intend to use four examples to let everyone feel that the sed command is really a "powerful" player.
For the first example, let's display the contents of lines 10 to 20 of the test file:
#We adopted the method of specifying the line range just mentioned [roc@roclinux ~]$ sed -n '10,20p' test
In the second example, we try to change all lowercase x characters in all lines starting with d or D to uppercase X characters:
[roc@roclinux ~]$ sed '/^[dD]/s/x/X/g' test
This usage is worth talking about. We use the syntax format of /AA/s/BB/CC/g in the command part, which means that we want to match lines with AA in the file and replace all BB in these lines with CC.
In the third example, we want to delete the last two characters of each line:
#The dot represents a single character, and two dots represent two single characters [roc@roclinux ~]$ sed 's/..$//' test
Someone may ask, why can't I use sed'/..$/d'test? Doesn't d mean delete? It is not possible to use d, because d means to delete the entire line content, not the characters. '/..$/d'It means matching all lines with two characters at the end, and then deleting the entire line. Obviously this is contrary to our original intention.
Fourth example, we want to remove the first two characters of each line:
[roc@roclinux ~]$ sed 's/..//' test
Through these four examples, I believe that everyone has a very intuitive understanding of the most common uses of the sed command. You might as well use this knowledge in your actual work.
Amazing uses of the & symbol
We still explain this knowledge point through a scenario.
#As usual, the content of the file is displayed first [roc@roclinux ~]$ cat mysed.txt Beijing London #We used the & symbol, please try to guess its function [roc@roclinux ~]$ sed 's/B .*/&2008/' mysed.txt Beijing2008 London
Without giving away the answer, the answer is revealed. The function of this command is to add the four characters 2008 to the end of the string containing 'B.*'. We use the & character in this command. In the sed command, it represents the "previously matched part", which of course is Beijing in our example!
Let’s use another example to strengthen everyone’s understanding of the ampersand:
#This example may be easier to understand [roc@roclinux 20160229]$ sed 's/Bei/&2008/' mysed.txt Bei2008jing London
The parentheses in sed have profound meanings
In the sed command, the parentheses '()' also have profound meaning. Let’s get straight to the point and let you see the power of parentheses through an example:
[roc@roclinux ~]$ echo "hello world" | sed 's/\(hello\).*/world \1/' world hello
We see that originally it was "hello world", but after sed processing, the output became "world hello".
This example uses the knowledge of parentheses, which we call "sed's pre-storage technology", that is, the content enclosed by "(" and ")" in the command will be temporarily stored in order and stored in \1 ,\2…inside. In this way you can use the '\N' form to call these pre-stored contents.
Let's continue to look at an example. We hope to only add the 2008 string after the first and last Beijing in each line. The implication is that, in addition to the first and last 2008 in each line, the string in the middle of this line Don’t add 2008 after Beijing when it appears. This requirement is really complex and personalized, but the sed command can still meet it well:
#Look at the file content first, there are 4 Beijings in the first line [roc@roclinux ~]$ cat mysed.txt Beijing Beijing Beijing London London London London #The effect is achieved, but the command is really complicated [roc@ roclinux ~]$ sed 's/\(Beijing\)\(.*\)\(Beijing\)/\12008\2\32008/' mysed.txt Beijing2008 Beijing Beijing2008 London London London London
This command is indeed complex enough, and in popular language, it is "sufficiently sadistic". In this example, we used pre-storage technology again to store three pieces of content, representing the first Beijing, the middle content, and the last Beijing. For \1sum \3, we append the string 2008 after it.
Smarter targeting of row ranges
Practice is the best way to learn knowledge. I believe that after reading this example, you will understand how to better position the row range:
#Show the file content [roc@roclinux ~]$ cat mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 Beijing 2007 #We want to show the content between the lines matching 2005 and 2007 [roc@ roclinux ~]$ sed -n '/2005/,/2007/p' mysed.txt Beijing 2005 Beijing 2006 Beijing 2007
We use /2005/ to match the first line of the line range and /2008/ to match the last line of the line range. It can be seen that when matching the last line, as long as it encounters the first line that meets the requirements, it will stop and will not continue to match backwards. Therefore, the sed command only matched the first 2007, but not the second 2007.
Use the -e option to set multiple commands
Remember the command part, now there is good news for you, that is, the sed command can contain more than one command. If you want to include multiple commands, just add an -e option in front of each command.
#We set two commands through 2 -e options [roc@roclinux ~]$ sed -n -e '1,2p' -e '4p' mysed.txt Beijing 2003 Beijing 2004 Beijing 2006
There is one thing worth paying attention to, that is, -ethe command content must be immediately followed by the option, and no other options are allowed. -eOptions support setting multiple commands, which is originally a good thing and allows us to achieve some replacement effects more conveniently. However, this also brings us happy troubles. If we set many commands, what is their execution order?
If this is not understood, -ethe options may bring confusion rather than convenience. Let’s take a look at the following example:
#First look at the content of the file [roc@roclinux ~]$ cat mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 #We set up two commands [roc@roclinux ~]$ sed -e 's/ Beijing/London/g' -e '/Beijing/d' mysed.txt London 2003 London 2004 London 2005 London 2006 London 2007 London 2008
The former command means to replace Beijing with London, and the latter command means to delete the lines containing the Beijing string, but the final result is that all lines are output, and the deleted lines are not found. This is because the first command has replaced Beijing with London, so the second command cannot find Beijing.
Let’s reverse the position of the command in the above example and see the effect:
#We first specify the deletion action, and then specify the replacement action [roc@roclinux 20160229]$ sed -e '/Beijing/d' -e 's/Beijing/London/g' mysed.txt [roc@roclinux 20160229]$
Through these two small examples, we can clearly see that multiple commands are executed according to the order in the command.
Set the command file with the -f option
If the command part of your sed command is very long, you can write the contents to a separate file and then use the -foption to specify this file as the command part of our sed command:
#This is the file we wrote in advance [roc@roclinux ~]$ cat callsed /2004/,/2006/p #We use the -f option to specify the command file [roc@roclinux ~]$ sed -n -f callsed mysed .txt Beijing 2004 Beijing 2005 Beijing 2006
It’s easy to understand. -fThe options are not difficult, and I will use them frequently, because I will save some of the more commonly used matching rules in separate files, so I don’t have to worry about memorizing them.
content insertion
The sed command is far more powerful than you think. It can not only process the content of this line, but also insert content after this line:
#We will save the content to be inserted into a separate file [roc@roclinux ~]$ cat ins.txt ====China==== #Show the file we want to process [roc@roclinux ~]$ cat mysed .txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 #Look, we use r to implement insertion [roc@roclinux ~]$ sed '/2005/r ins.txt' mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 = ===China==== Beijing 2006 Beijing 2007 Beijing 2008
As can be seen from the effect, we inserted the contents of the ins.txt file in the line below the line containing the 2005 string in the file.
In addition to inserting by specifying a file, you can also use a\ to insert specific content "below" a specific line:
#File content [roc@roclinux ~]$ cat new.txt Beijing 2004 Beijing 2005 Beijing 2006 #We hope to insert China in the next line of 2004 [roc@roclinux ~]$ sed '/2004/a\China' mysed.txt Beijing 2003 Beijing 2004 China Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008
As you can see, we a\can easily achieve this by just using and adding the content to be inserted.
Some students may ask, since content can be inserted below a line, can content be inserted above a line? The answer is of course, using i\actions:
[roc@roclinux ~]$ sed ‘/2004/i\China’ mysed.txt Beijing 2003 China Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008
Talk about y action
Before introducing the y action, let's first take a look at what effects it can achieve:
#Original file content [roc@roclinux ~]$ cat mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 #y is to replace before and after in character order [roc@roclinux 20160229]$ sed 'y/ei /ie/' mysed.txt Biejeng 2003 Biejeng 2004 Biejeng 2005 Biejeng 2006 Biejeng 2007 Biejeng 2008
This example is actually very clear, we want to interchange all e and i.
Some students will ask, what is the difference between y///and ? s///There are mainly two points:
- The syntax format of y is y/source/dest/, which means to replace the characters in source with the characters in dest. The syntax format of s is s/regexp/replacement/, which means that the content matched by the regular expression is replaced with the replacement part.
- y is just a simple word-for-word replacement, without a lot of tricks. s supports features such as the & symbol and pre-storage, which can achieve more flexible replacement effects.
At this time, some GEEKs may think of a situation, that is, what effect will y/ee/ei/ have? Because there are two same characters here, let’s take a look at it through an example:
[roc@roclinux 20160229]$ sed 'y/ee/ie/' mysed.txt Biijing 2003 Biijing 2004 Biijing 2005 Biijing 2006 Biijing 2007 Biijing 2008
As you can see, if there are repeated characters in the source part, only the first occurrence of the replacement will have an effect, and the subsequent ones will not. Perhaps the following example will be clearer:
#Original file content [roc@roclinux ~]$ cat mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 #In the replacement of iji to iba, only j to b has the effect [roc@roclinux 20160229]$ sed 'y/iji/iba/' mysed.txt Beibing 2003 Beibing 2004 Beibing 2005 Beibing 2006 Beibing 2007 Beibing 2008
Control the downward movement of rows through n actions
Sometimes we want to achieve the effect of interlaced processing, for example, we only need to make a certain replacement for even-numbered lines. At this time, we need the help of n action:
#Original content [roc@roclinux ~]$ cat mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 n;y/eijng/EIJNG/;}' mysed.txt Beijing 2003 BEIJING 2004 Beijing 2005 BEIJING 2006 Beijing 2007 BEIJING 2008
You will find that the uppercase BEIJING appears on alternate lines. This is nhow the option works. Its real function is to put the next line of content into the processing cache, so that the current line can avoid the replacement action. Isn't it a bit like using the left and right keys to avoid the BOSS when playing games as a child? Big move, haha.
Write the specified line to a specific file
The article is coming to an end. Let us finally teach you a very practical action, which is the w action. It can write the matched content to another file, which is used to filter and save the content:
#Save the lines containing 2004, 2005, and 2006 into the new.txt file [roc@roclinux ~]$ sed '/200[4-6]/w new.txt' mysed.txt Beijing 2003 Beijing 2004 Beijing 2005 Beijing 2006 Beijing 2007 Beijing 2008 #The content we want has arrived in the bowl [roc@roclinux ~]$ cat new.txt Beijing 2004 Beijing 2005 Beijing 2006