10.1 cut
The job of cut is to "cut", specifically, it is responsible for cutting data in the file. The cut command cuts bytes, characters, and fields from each line of a file and outputs them.
1. Basic usage
cut [option parameter] filename
Description: The default delimiter is tab
2. Option parameter description
Table 1-55
| option parameter |
Function |
| -f |
Column number, which column to extract |
| -d |
Delimiter, split columns according to the specified delimiter |
3. Case Practice
(0) Data preparation
[atguigu@hadoop101 datas]$ touch cut.txt
[atguigu@hadoop101 datas]$ vim cut.txt
dong shen
guan zhen
wo wo
so be it
the the
(1) Cut the first column of cut.txt
[atguigu@hadoop101 datas]$ cut -d " " -f 1 cut.txt
dong
guan
wo
to
the
(2) Cut the second and third columns of cut.txt
[atguigu@hadoop101 datas]$ cut -d " " -f 2,3 cut.txt
shen
zhen
wo
to
the
(3) Cut out guan in the cut.txt file
[atguigu@hadoop101 datas]$ cat cut.txt | grep "guan" | cut -d " " -f 1
guan
(4) Select the system PATH variable value, all paths after the second ":" start:
[atguigu@hadoop101 datas]$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/atguigu/bin
[atguigu@hadoop102 datas]$ echo $PATH | cut -d: -f 2-
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/atguigu/bin
(5) IP address printed after cutting ifconfig
[atguigu@hadoop101 datas]$ ifconfig eth0 | grep "inet addr" | cut -d: -f 2 | cut -d" " -f1
192.168.1.102
10.2 sed
sed is a stream editor that processes content one line at a time. When processing, the currently processed line is stored in a temporary buffer, which is called "pattern space", and then the content in the buffer is processed with the sed command. After the processing is completed, the content of the buffer is sent to the screen. Then process the next line, and repeat until the end of the file. The file contents are not changed unless you use redirected storage output.
- basic usage
sed [options] 'command' filename
- Option parameter description
Table 1-56
| option parameter |
Function |
| -e |
Edit sed actions directly in command line mode. |
- Command function description
Table 1-57
| Order |
Functional description |
| a |
Added, a string can be connected after a, and it will appear on the next line |
| d |
delete |
| s |
find and replace |
- Case Practice
(0) Data preparation
[atguigu@hadoop102 datas]$ touch sed.txt
[atguigu@hadoop102 datas]$ vim sed.txt
dong shen
guan zhen
wo wo
so be it
the the
(1) Insert the word "mei nv" into the second line of sed.txt and print it.
[atguigu@hadoop102 data]$ seed '2nd mei nv' sed.txt
dong shen
guan zhen
with nv
wo wo
so be it
the the
[atguigu@hadoop102 datas]$ cat sed.txt
dong shen
guan zhen
wo wo
so be it
the the
Note: the file has not changed
(2) Delete all lines containing wo in the sed.txt file
[atguigu@hadoop102 data]$ sed '/wo/d' sed.txt
dong shen
guan zhen
so be it
the the
(3) Replace wo with ni in the sed.txt file
[atguigu@hadoop102 data]$ sed 's/wo/ni/g' sed.txt
dong shen
guan zhen
ni ni
so be it
the the
Note: 'g' means global , replace all
(4) Delete the second line in the sed.txt file and replace wo with ni
[atguigu@hadoop102 data]$ sed -e '2d' -e 's/wo/ni/g' sed.txt
dong shen
ni ni
so be it
the the
10.3 awk
A powerful text analysis tool that reads files line by line, slices each line with spaces as the default delimiter, and then analyzes and processes the cut parts.
- basic usage
awk [option parameter] 'pattern1{action1} pattern2{action2}...' filename
pattern: Indicates what AWK is looking for in the data, which is the matching pattern
action: A sequence of commands to execute when a match is found
- Option parameter description
Table 1-55
| option parameter |
Function |
| -F |
Specify input file fold delimiter |
| -v |
Assign a value to a user-defined variable |
- Case Practice
(0) Data preparation
[atguigu@hadoop102 datas]$ sudo cp /etc/passwd ./
(1) Search for all lines starting with the root keyword in the passwd file, and output column 7 of the line.
[atguigu@hadoop102 datas]$ awk -F: '/^root/{print $7}' passwd
/bin/bash
(2) Search for all lines starting with the root keyword in the passwd file, and output the first and seventh columns of the line, separated by "," in the middle.
[atguigu@hadoop102 datas]$ awk -F: '/^root/{print $1","$7}' passwd
root,/bin/bash
Note: only the lines that match the pattern will execute the action
(3) Only the first and seventh columns of /etc/passwd are displayed, separated by commas, and the column name user is added in front of all lines, and the shell adds "dahaige, /bin/zuishuai" to the last line.
[atguigu@hadoop102 datas]$ awk -F : 'BEGIN{print "user, shell"} {print $1","$7} END{print "dahaige,/bin/zuishuai"}' passwd
user, shell
root,/bin/bash
bin,/sbin/nologin
。。。
atguigu,/bin/bash
dahaige,/bin/zuishuai
Note: BEGIN is executed before all data read rows; END is executed after all data is executed.
(4) Increase the user id in the passwd file by 1 and output
[ atguigu@hadoop102 data] $awk -vi=1 -F: '{print $3+i}' passwd
1
2
3
4
- Awk's built-in variables
Table 1-56
| variable |
illustrate |
| FILENAME |
file name |
| NR |
Number of records read |
| NF |
The number of domains in the browse record (after cutting, the number of columns) |
- Case Practice
(1) Statistics passwd file name, line number of each line, number of columns of each line
[atguigu@hadoop102 datas]$ awk -F: '{print "filename:" FILENAME ", linenumber:" NR ",columns:" NF}' passwd
filename:passwd, linenumber:1,columns:7
filename:passwd, linenumber:2,columns:7
filename:passwd, linenumber:3,columns:7
(2) Cut IP
[atguigu@hadoop102 datas]$ ifconfig eth0 | grep "inet addr" | awk -F: '{print $2}' | awk -F " " '{print $1}'
192.168.1.102
(3) Query the line number of the blank line in sed.txt
[atguigu@hadoop102 datas]$ awk '/^$/{print NR}' sed.txt
5
10.4 sort
The sort command is very useful in Linux, it sorts files and outputs the sorted results to standard output.
- basic grammar
sort(option)(argument)
Table 1-57
| options |
illustrate |
| -n |
Sort by numerical value |
| -r |
sort in reverse order |
| -t |
Set the delimiter character used when sorting |
| -k |
Specify the column to be sorted |
Parameters: Specify the list of files to be sorted
2. Case study
(0) Data preparation
[atguigu@hadoop102 datas]$ touch sort.sh
[atguigu@hadoop102 datas]$ vim sort.sh
bb:40:5.4
bd:20:4.2
xz:50:2.3
cls:10:3.5
ss:30:1.6
(1) Sort in reverse order according to the third column separated by ":".
[atguigu@hadoop102 datas]$ sort -t : -nrk 3 sort.sh
bb:40:5.4
bd:20:4.2
cls:10:3.5
xz:50:2.3
ss:30:1.6