[Comprehensive explanation of Linux commands] 081.sed: A powerful streaming text editor

sed

Powerful streaming text editor

Additional information

sed is a stream editor. It is a very important tool in text processing. It can be used perfectly with regular expressions and has extraordinary functions. During processing, the currently processed line is stored in a temporary buffer, called "pattern space", and then the sed command is used to process the contents of the buffer. After the processing is completed, the contents of the buffer are sent to the screen. Then process the next line, and repeat until the end of the file. The file contents are not changed unless you use redirection to store the output. Sed is mainly used to automatically edit one or more files; simplify repeated operations on files; write conversion programs, etc.

sed options, commands, replacement tags

Command format

sed [options] 'command' file(s)
sed [options] -f scriptfile file(s)

options

• -e

parameter

• File: Specify a list of text files to be processed.

sed command

• a\ # Insert text below the current line.
• i\ # Insert text above the current line.
• c\ # Change the selected line to new text.
• d # Delete, delete the selected row.
• D # Delete the first line of the template block.
• s #Replace specified characters
• h #Copy the contents of the template block to the buffer in memory.
• H # Append the contents of the template block to the buffer in memory.
• g # Get the contents of the memory buffer and replace the text in the current template block.
• G # Get the contents of the memory buffer and append it to the text of the current template block.
• l # List cannot print a list of characters.
• n # Read the next input line and use the next command to process the new line instead of the first command.
• N # Append the next input line after the template block and embed a new line between them, changing the current line number.
• p # Print the lines of the template block.
• P # (uppercase) prints the first line of the template block.
• q # Quit Sed.
• b <label> # Branch to the labeled place in the script, or branch to the end of the script if the branch does not exist.
• r <file> # Read lines from file.
• t <label> #if branch, starting from the last line, once the condition is met or the T, t command will cause the branch to the command with the label, or to the end of the script.
• T <label> # Error branch, starting from the last line. Once an error or T, t command occurs, it will cause the branch to the command with the label, or to the end of the script.
• w <file> # Write and append the template block to the end of file.
• W <file> # Write and append the first line of the template block to the end of file.
• ! # Indicates that the following commands will take effect on all unselected lines.
• = # Print the current line number.

• # Extend the comment to the next newline character.

sed replace tag

• g # means full replacement within the line.
• p # means print line.
• w # means writing lines to a file.
• x # means swapping the text in the template block and the text in the buffer.
• y # means translating one character into another character (but not used in regular expressions)
• \1 # substring matching token
• & # Matched string tag

sed metacharacter set

• ^ # Match the beginning of the line, such as: /^sed/ matches all lines starting with sed.
• $ # Matches the end of a line, such as: /sed$/ matches all lines ending with sed.
• . # Matches any character that is not a newline character, such as: /sd/ matches s followed by any character, and finally d.
• * # Match 0 or more characters, such as: /*sed/ Matches all lines whose template is one or more spaces followed by sed.
• [] # Matches characters within a specified range, such as /[sS]ed/ matches sed and Sed.
• [^] # Matches a character that is not within the specified range, such as: /[^A-RT-Z]ed/ Matches lines starting with a letter that does not contain AR and TZ, followed by ed.
• (...) # Match substrings and save matching characters, such as s/(love)able/\1rs, loveable is replaced by lovers.
• & # Save search characters to replace other characters, such as s/love/ & /, love becomes love .
• < # Matches the beginning of a word, such as:/<love/ Matches lines containing words starting with love.
• > # Match the end of a word, such as /love>/ Matches lines containing words ending with love.
• x{m} # Repeat character x, m times, such as: /0{5}/ matches lines containing 5 zeros.
• x{m,} # Repeat the character x at least m times, such as: /0{5,}/ matches lines with at least 5 0s.
• x{m,n} # Repeat the character x at least m times and no more than n times, such as: /0{5,10}/ matches lines with 5 to 10 zeros.

sed usage examples

Substitution operation: s command

Replace strings in text:

sed 's/book/books/' file

The -n option is used with the p command to print only those lines where substitutions occur:

sed -n ‘s/test/TEST/p’ file

Directly edit the file option -i, which will replace all books matching each line in the file file with books:

sed -i 's/book/books/g' file

full replacement mark g

Using the suffix /g flag replaces all matches on each line:

sed 's/book/books/g' file

When you need to start replacing from the Nth match, you can use /Ng:

echo sksksksksksk | sed 's/sk/SK/2g'
skSKSKSKSKSK

echo sksksksksksk | sed 's/sk/SK/3g'
skskSKSKSKSK

echo sksksksksksk | sed 's/sk/SK/4g'
skskskSKSKSK

delimiter

The character / in the above command is used as a delimiter in sed, and any delimiter can also be used:

sed 's:test:TEXT:g'
sed 's|test|TEXT|g'

Delimiters need to be escaped when they appear inside styles:

sed 's/\/bin/\/usr\/local\/bin/g'

Delete operation: d command

Remove blank lines:

sed '/^$/d' file

Delete line 2 of the file:

sed '2d' file

Delete all lines from line 2 to the end of the file:

sed '2,$d' file

Delete the last line of the file:

sed '$d' file

Delete all lines starting with test in the file:

sed '/^test/'d file

Matched string tag &

The regular expression \w+ matches every word and replaces it with [&], where & corresponds to the previously matched word:

echo this is a test line | sed 's/\w\+/[&]/g'
[this] [is] [a] [test] [line]

All lines starting with 192.168.0.1 will be replaced with itself plus localhost:

sed 's/^192.168.0.1/&localhost/' file

192.168.0.1localhost

substring matches token \1

Matches part of a given pattern:

echo this is digit 7 in a number | sed 's/digit \([0-9]\)/\1/'
this is 7 in a number

The digit 7 in the command was replaced with 7. The substring matched by the pattern is 7, (...) is used to match substrings, the first matched substring is marked as \1, and so on, the second matched result is \2, for example:

echo aaa BBB | sed 's/\([a-z]\+\) \([A-Z]\+\)/\2 \1/'
BBB aaa

love is marked as 1, all loveable will be replaced by lovers, and printed out:

sed -n 's/\(love\)able/\1rs/p' file

Get the ip by replacing:

ifconfig ens32 | sed -n '/inet /p' | sed 's/inet \([0-9.]\+\).*/\1/'
192.168.75.126

Case conversion U/L

• \u: Convert first letter to uppercase
• \U: Convert all to uppercase
• \l: Convert the first letter to lowercase
• \L: Convert all to lowercase

Convert first letter to uppercase:

[root@node6 ~]# sed 's/^[a-z]\+/\u&/' passwd 
Root:x:0:0:root:/root:/bin/bash
Bin:x:1:1:bin:/bin:/sbin/nologin
Daemon:x:2:2:daemon:/sbin:/sbin/nologin
Adm:x:3:4:adm:/var/adm:/sbin/nologin
Lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
Sync:x:5:0:sync:/sbin:/bin/sync

All matched characters are converted to uppercase:

[root@node6 ~]# sed 's/^[a-z]\+/\U&/' passwd 
ROOT:x:0:0:root:/root:/bin/bash
BIN:x:1:1:bin:/bin:/sbin/nologin

Range of selected rows:, (comma)

All lines within the range determined by templates test and check are printed:

sed -n '/test/,/check/p' file

Print all lines starting at line 5 and ending with the first line containing test:

sed -n '5,/^test/p' file

For the lines between template test and west, replace the end of each line with the string aaa bbb:

sed '/test/,/west/s/$/aaa bbb/' file

Multi-point editing: e command

The -e option allows multiple commands to be executed on the same line:

sed -e '1,5d' -e 's/test/check/' file

The first command of the sed expression above deletes lines 1 to 5, and the second command replaces test with check. The order in which commands are executed affects the results. If both commands are substitution commands, then the first substitution command will affect the results of the second substitution command.

The equivalent command to -e is --expression:

sed --expression='s/test/check/' --expression='/love/d' file

Read from file: r command

The contents of file are read in and displayed after the line matching test. If multiple lines are matched, the contents of file will be displayed below all matching lines:

sed '/test/r file' filename

Write to file: w command

In the example all lines containing test are written to file:

sed -n '/test/w file' example

Append (below the line): a\command

Append this is a test line to the line starting with test:

sed '/^test/a\this is a test line' file

Insert this is a test line after line 2 in the test.conf file:

sed -i '2a\this is a test line' test.conf

Insert (on line): i\command

Append this is a test line to the line starting with test:

sed '/^test/i\this is a test line' file

Insert this is a test line before line 5 of the test.conf file:

sed -i '5i\this is a test line' test.conf

Replace specified line: c\ command

Replace the lines starting with root with new content:

[root@node6 ~]# sed '/^root/c this is new line!' passwd 
this is new line!
bin:x:1:1:bin:/bin:/sbin/nologin
Daemon:x:2:2:daemon:/sbin:/sbin/nologin
Adm:x:3:4:adm:/var/adm:/sbin/nologin
Lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
Sync:x:5:0:sync:/sbin:/bin/sync

If you are replacing a specified range, please note that sed does not replace each line, but replaces the entire range as a whole:

[root@node6 ~]# nl passwd | sed '1,5c\   this is dangerous!'
     this is dangerous!
     6	sync:x:5:0:sync:/sbin:/bin/sync
     7	shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown

If you want to uniformly replace the first to fifth lines with the same content, you can use the following command:

[root@node5 ~]# sed '1{:a;s/.*/lutxixia/;n;6!ba}' passwd 
lutxixia
lutxixia
lutxixia
lutxixia
lutxixia
sync:x:5:0:sync:/sbin:/bin/sync

in:

• :a is to set a loop label
• s/.*/lutxixia/ is to replace the matched content of each line with lutxixia characters
• n is to read the next line
• 6! is to read the sixth line to exit the loop, terminate the operation, if not, continue the loop.
• ba is to jump to a and continue the loop if the sixth line is not reached

Next: n command

If test is matched, move to the next line after the matching line, replace aa in this line with bb, print the line, and continue:

sed '/test/{ n; s/aa/bb/; }' file

Transformation: y command

Convert all abcde in lines 1 to 10 to uppercase. Note that regular expression metacharacters cannot use this command:

sed '1,10y/abcde/ABCDE/' file

Quit: q command

After printing the first 10 lines, exit sed:

sed '10q' file

Until the first match is found, exit sed:

[root@node4 ~]# sed  '/nginx/q' nginx.yml 
---
- hosts: nginx

Hold and get: h command and G command

When sed processes a file, each line is saved in a temporary buffer called the pattern space. Unless the line is deleted or the output is cancelled, all processed lines will be printed on the screen. Then the pattern space is cleared and a new line is stored for processing.

sed -e '/test/h' -e '$G' file

In this example, after the line matching test is found, it will be stored in the pattern space. The h command will copy it and store it in a special buffer called the holding buffer. The second statement means that when the last line is reached, the G command takes out the line held in the buffer, puts it back into the pattern space, and appends it to the end of the line that now exists in the pattern space. In this example it is appended to the last line. Simply put, any line containing test is copied and appended to the end of the file.

Keep and swap: h command and x command

Swap the pattern space and hold the contents of the buffer. That is, swap the lines containing test and check:

sed -e '/test/h' -e '/check/x' file

Script scriptfile

The sed script is a sed command list. When starting Sed, the -f option is used to guide the script file name. Sed is very picky about the commands entered in the script. There cannot be any whitespace or text at the end of the command. If there are multiple commands on a line, they must be separated by semicolons. Lines starting with # are comment lines and cannot span lines.

sed [options] -f scriptfile file(s)

Print odd or even lines

method 1:

sed -n 'p;n' test.txt  #奇数行
sed -n 'n;p' test.txt  #偶数行

Method 2:

sed -n '1~2p' test.txt  #奇数行
sed -n '2~2p' test.txt  #偶数行

Print the next line of the matching string

grep -A 1 SCC URFILE
sed -n '/SCC/{n;p}' URFILE
awk '/SCC/{getline; print}' URFILE

Learn from scratchpython

[Learn python from scratch] 92. Use Python’s requests library to send HTTP requests and process responses
[Learn python from scratch] 91. Use decorators and dictionaries to manage request paths in a simple web application
[Learn python from scratch] 93. Use dictionary management Request path
[Learn python from scratch] 89. Use WSGI to build a simple and efficient Web server
[Learn python from scratch] 88. Detailed explanation of WSGI interface: realize simple and efficient web development
[Learn python from scratch] 87. Manually build an HTTP server in Python Implementation and multi-threaded concurrent processing
[Learn python from scratch] 86. In-depth understanding of the HTTP protocol and its role in browser and server communication
[Learn python from scratch] 85. Application of parallel computing technology in Python process pool
[Learn python from scratch] ] 84. In-depth understanding of threads and processes
[Learn python from scratch] 83. Python multi-process programming and the use of process pools
[Learn python from scratch] 82. Chat program implementation based on multi-threading
[Learn python from scratch] 81. Python more Application of thread communication and queue
[Learn python from scratch] 80. Thread access to global variables and thread safety issues
[Learn python from scratch] 79. Thread access to global variables and thread safety issues
[Learn python from scratch] 78. File download case
[ Learn python from scratch] 77. TCP server programming and precautions
[learn python from scratch] 76. Server and client: key components of network communication
[learn python from scratch] 75. TCP protocol: reliable connection-oriented transmission layer communication protocol
[Learn python from scratch] 74. UDP network program: Detailed explanation of port issues and binding information
[Learn python from scratch] 73. UDP network program - sending data
[Learn python from scratch] 72. In-depth understanding of Socket communication and creation of sockets Method
[Learn python from scratch] 71. Network ports and their functions
[Learn python from scratch] 70. Network communication methods and their applications: from direct communication to routers to connect multiple networks
[Learn python from scratch] 69. Network communication and IP address classification analysis
[Learn python from scratch] 68. Greedy and non-greedy modes in Python regular expressions
[Learn python from scratch] 67. The re module in Python: regular replacement and advanced matching technology
[Learn python from scratch] 66 .In-depth understanding of regular expressions: a powerful tool for pattern matching and text processing
[Learn python from scratch] 65. Detailed explanation of Python regular expression modifiers and their applications
[Learn python from scratch] 64. The re.compile method in Python regular expressions Detailed explanation of usage
[Learn python from scratch] 63. Introduction to the re.Match class and its attributes and methods in regular expressions
[Learn python from scratch] 62. Python regular expressions: a powerful string matching tool
[Learn python from scratch] 61. Detailed explanation and application examples of property attributes in Python
[Learn python from scratch] 60. Exploration generator: a flexible tool for iteration
[Learn python from scratch] 59. Iterator: An efficient tool for optimizing data traversal
[Learn python from scratch] 58. Custom exceptions in Python and methods of raising exceptions
[Learn python from scratch] 57. Use the with keyword in Python to correctly close resources
[Learn python from scratch] 56. The importance and application of exception handling in programming
[Learn python from scratch] 55. Serialization and sum in Python Deserialization, application of JSON and pickle modules
[Learn python from scratch] 54. Writing data in memory
[Learn python from scratch] 53. CSV files and Python’s CSV module
[Learn python from scratch] 52. Reading and writing files - Python file operation guide
[Learn python from scratch] 51. Opening and closing files and their applications in Python
[Learn python from scratch] 49. Object-related built-in functions in Python and their usage
[Learn python from scratch] 48 .Detailed explanation of inheritance and multiple inheritance in Python
[Learn python from scratch] 47. The concept and basic use of inheritance in object-oriented programming
[Learn python from scratch] 46. Analysis of __new__ and __init__ methods and singletons in Python Design Patterns
[Learn python from scratch] 45. Class methods and static methods in Python
[Learn python from scratch] 44. Private properties and methods in object-oriented programming
[Learn python from scratch] 43. Examples in Python object-oriented programming Properties and class attributes
[Learn python from scratch] 42. Built-in properties and methods in Python
[Learn python from scratch] 41. python magic method (2)
[Learn python from scratch] 40. python magic method (1)
[Learn python from scratch] 39. Basic object-oriented syntax and application examples
[Learn python from scratch] 38. How to use and import Python packages
[Learn python from scratch] 37. The use and precautions of Python custom modules
[Learn python from scratch] Learn python] 36. Methods and techniques of using pip for third-party package management in Python
[Learn python from scratch] 35. Common Python system modules and their usage
[Learn python from scratch] 34. Detailed explanation of the import and use of Python modules
[ Learn python from scratch] 33. The role of decorators (2)
[Learn python from scratch] 32. The role of decorators (1)
[Learn python from scratch] 31. In-depth understanding of higher-order functions and closures in Python
[From Learn python from scratch】30. In-depth understanding of recursive functions and anonymous functions
【learn python from scratch】29. "Detailed explanation of function parameters" - understand the different uses of Python function parameters
【learn python from scratch】28. Local variables and global variables in Python Variables
[Learn python from scratch] 27. The use of Python functions and nested calls
[Learn python from scratch] 25. Functions: a tool to improve the efficiency of code writing
[Learn python from scratch] 24. String operations and traversal methods in Python
[Learn python from scratch] 23. How to use sets (set) and common operations in Python
[Learn python from scratch] 22. Add, delete, modify, and query dictionary variables in Python
[Learn python from scratch] 21. Python tuples and dictionaries
[Learn python from scratch] 20. Python list operation skills and examples
[Learn python from scratch] 19. Applications of looping through lists and list nesting
[Learn python from scratch] 18. Detailed explanation of the basic operations of Python lists (1)
[From Learning python from scratch】 17. The format method of Python strings (2)
【Learning python from scratch】 16. The format method of Python strings (1)
【Learning python from scratch】 15. In-depth understanding of strings and character set encoding
【From Learning python from scratch】14. Common operations on Python strings (2)
【Learning python from scratch】13. Common operations on Python strings (1)
【Learning python from scratch】12. Python string operations and applications
【Learning python from scratch】 11. Python loop statements and control flow
[Learn python from scratch] 10. Detailed explanation of Python conditional statements and if nesting
[Learn python from scratch] 09. Conditional judgment statements in Python
[Learn python from scratch] 08. Python understands bit operations operator, operator priority
[Learn python from scratch] 07. Detailed explanation of Python operators: assignment, comparison and logical operators
[Learn python from scratch] 06. Use arithmetic operators in Python for calculations and string concatenation
[Learn from scratch] python ] 05. Output and input in Python
[Learn python from scratch] 04. Basics of Python programming: variables, data types and identifiers
[Learn python from scratch] 03. Python interactive programming and detailed explanation of comments
[Learn python from scratch] 02. Introduction to development tools
[Learn python from scratch] 01. Install and configure python