Detailed explanation of Linux basic commands and tools

1. grep search characters

The grep command is used to perform keyword searches in files and display the results of matches. Some common options:

• -c only show the number of lines found
• -i ignore case
• -n display line number
• -v reverse selection - list only lines without keywords. v is an abbreviation for invert.
• -r recursively search file directories
• -C n print n lines before and after the matching line

(1) Search in the specified file and find the login keyword:

grep battery ./shell/battery.sh

(2) When searching for multiple files, wildcards can be used. For example, in a file ending with sh, search for lines containing battery:

grep battery *.sh

(3) Recursively search all files in the directory, search all files in the msg_server directory, and print out the line containing battery:

grep battery -r msg_server

(4) Reverse lookup, look for lines that do not contain battery in the file:

grep -v battery ImUser.cpp

(5) Find the line containing battery in the file and print the line number:

grep -n battery ./shell/battery.sh

(6) Find the line containing login in the file, print the line number, and display the 3 lines before and after.

grep login -C 3 -n test.cpp

(7) Find out the line containing login in the file, print the line number, and display 3 lines before and after, and ignore case:

grep login -n -C 3 -i test.cpp

Two, find find files

Find the location of the file through the file name, and the file name search supports fuzzy matching.

find  [指定查找目录]  [查找规则]  [查找完后执行的action] 

Common operations:

find . -name FILE_NAME
find . -iname FILE_NAME #忽略文件名称大小写
find /etc -maxdepth 1 -name passwd     ##查找/etc/下名称中带有passwd的文件，查找一层
find /mnt -size 20K       ##查找/mnt文件大小近似20k的文件
find /mnt -size +20K      ##查找/mnt文件大小大于20k的文件
find /mnt -size -20K      ##查找/mnt文件大小小于20k的文件
find /etc -maxdepth 2 -mindepth 2 -name .conf  ##查找/etc/下名称中带有.conf的文件，且只查找第二层
find /mnt -type d         ##按type查找/mnt中目录
find /mnt -type f         ##按type查找/mnt中文件
find /mnt -cmin 10        ##查找/mnt中十分钟左右修改的
find /mnt -cmin +10       ##查找/mnt中十分钟以上修改的
find /mnt -cmin -10       ##查找/mnt中十分钟以内修改的
find /mnt -ctime 10       ##查找/mnt中十天左右修改的
find /mnt -ctime +10      ##查找/mnt中十天以上修改的
find /mnt -ctime -10      ##查找/mnt中十天以内修改的

3. ls display files

Parameter role

• -t View the latest modification time
• -l display one entry per line
• -h can be combined to display the GB, MB, etc. of the file (human);
• -R display recursively
• -n display group id and gid

(1) Sort by the time of the latest modification, and the newly modified ones will be displayed first.

ls -lt

(2) Sort by the latest modification time, the newly modified ones will be displayed in front, and the file information of subdirectories will be displayed.

ls -ltR

(3) Display file size in units.

ls -lh 

Fourth, the wc command calculates the number of words

The wc command is used to count words. We can use the wc command to calculate the number of bytes, words, or columns of the file. If the file name is not specified, or the given file name is "-", the wc command will read data from the standard input device.

grammar:

wc [-clw][--help][--version][文件...]

parameter:

• -c or --bytes or --chars: only display the number of Bytes.
• -l or --lines: Only display the number of lines.
• -w or --words: Only display the word count.
• –help: Online help.
• –version: Display version information.

Example:

$ wc ./shell/battery.sh 
 125  333 3157 ./shell/battery.sh

The number of lines is 125, the number of words is 333, and the number of bytes is 3157.

Five, uptime machine startup time + load

Check the startup time of the machine, the logged-in user, the average load, etc. It is usually used in online emergency or technical research to determine the restart time of the operating system.

Example:

$ uptime 
 15:29:42 up 9 days,  5:58,  1 user,  load average: 0.00, 0.00, 0.00

From the above output, you can see the following information:

• Current time: 15:29:42
• Time the system has been running: 9 days, 5 hours and 58 minutes.
• Previous online users: 1 user, which is the total number of connections, not the number of different users (opening a terminal connection counts as one user).
• System load average: 0.00, 0.00, 0.00, which are the system load conditions of the last 1 minute, 5 minutes, and 15 minutes.

The load average of a system is the average number of processes running in the queue during a specific time interval. If a process satisfies the following conditions, it will be placed in the run queue.

• It is not waiting for the result of the I/O operation.
• It does not actively enter the waiting state (that is, does not call the 'wait' related system API).
• is not stopped (eg: waiting for termination).

If the number of current active processes per CPU core is not greater than 3, then the performance of the system can still be supported.

If the number of tasks per CPU core is greater than 5, then there is a serious problem with the performance of this machine.

If your linux host is a dual-core CPU, when the Load Average is 6, it means that the machine has been fully used.

Load description (for single-core case, multiplied by the number of cores if not single-core):

• load<1: no waiting
• load==1: The system has no additional resources to run more processes
• load>1: processes are blocked waiting for resources

Notice:

• When load < 0.7: the system is very idle, consider deploying more services
• 0.7 < load < 1: the system is in good condition
• When load == 1: the system is about to deal with it, so quickly find the reason
• When load > 5: the system is already very busy

Problems explained by different load values:

• 1 minute load > 5, 5 minutes load < 3, 15 minutes load < 1, busy in the short term, idle in the medium and long term, the initial judgment is a jitter or a precursor to congestion.
• 1 minute load >5, 5 minutes load >3, 15 minutes load <1, busy in the short term and tense in the medium term, it is likely to be the beginning of a congestion.
• 1 minute load > 5, 5 minutes load > 5, 15 minutes load > 5, short, medium and long term are busy, the system is congested.
• Load <1 in 1 minute, Load >3 in 5 minutes, load >5 in 15 minutes, idle in the short term, busy in the medium and long term, don’t be nervous, the system congestion is getting better.

Replenish:

• View cpu information: cat /proc/cpuinfo.
• Directly obtain the number of cpu cores: grep 'model name' /proc/cpuinfo | wc -l .

Six, ulimit user resources

The Linux system limits the maximum number of processes and the maximum number of open file handles for each logged-in user. In order to improve performance, you can set the maximum number of processes and the maximum number of open file handles for each user according to the specific conditions of hardware resources.

(1) ulimit -a can be used to display the current various system restrictions on the user's use of resources:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31574
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31574
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

(2) Set the maximum number of processes for the user:

ulimit -u 1024 

(3) Set the maximum number of file handles that the user can open:

ulimit -n 65530 

七、curl http

Since current online services mostly use RESTful-style APIs, integration testing requires HTTP calls to check whether the returned results meet expectations. The curl command is of course the preferred testing method.

How to use:

curl -i "http://www.sina.com" # 打印请求响应头信息
curl -I "http://www.sina.com" # 仅返回http头
curl -v "http://www.sina.com" # 打印更多的调试信息
curl -verbose "http://www.sina.com" # 打印更多的调试信息
curl -d 'abc=def' "http://www.sina.com" # 使用post方法提交http请求
curl -sw '%{http_code}' "http://www.sina.com" # 打印http响应码

Eight, scp remote copy

The abbreviation of secure copy, scp is a secure remote file copy command based on ssh login under the linux system.

The scp command is a powerful file transfer command in the Linux system. It can realize two-way file transfer from local to remote and from remote to local. It is very convenient to use. It is often used to download some online files to Check locally, or upload local modifications to the server.

grammar:

scp [option] username@IP:filepath savepath

premise:

sudo apt-get install openssh-server

How to use:

# 下载192.168.11.59的文件
scp [email protected]:~/workspace/battery.sh .
# 上传文件到122.152.222.180
scp mi9.aac [email protected]:~/workspace/
# 下载test整个目录到本地
scp -r [email protected]:~/workspace/ .
# 上传本地test整个目录到192.168.11.59
scp -r test [email protected]:~/workspace/

Nine, dos2unix and unix2dos

It is used to convert the newline characters between Windows and UNIX. Usually, scripts and configurations developed on Windows systems need to be converted under UNIX systems.

How to use:

dos2unix test.txt 
unix2dos test.txt 
#转换整个目录 
find . -type f -exec dos2unix {
    
    } \; 
find ./ -type f # 此命令是显示当前目录下所有的文件

Ten, sed line processing

Command format 1:

sed 's/原字符串/新字符串/' 文件 

Command format 2:

sed 's/原字符串/新字符串/g' 文件 

The difference between the two command formats is whether there is a "g". No "g" means to replace only the first matched string, "g" means to replace all matching strings, "g" can be considered as the abbreviation of "global" (global), there is no "global" " Don't replace all at the end.

The sed command is used to modify the text content in batches, such as replacing an ip in the configuration in batches.

When the sed command is processed, it will first read a line, store the currently processed line in a temporary buffer, and print it to the screen after processing the contents of the buffer. Then read the next line and execute the next cycle. Repeatedly until the end of the file.

10.1. Simple mode

A simple sed command consists of three main parts: arguments, scope, and operation. The file to be operated can be directly hung at the end of the command line.

sed -n '2,5 p' filename.txt
# -n是参数
# 2,5 是范围
# p 是操作
# filename.txt是文件

(1) Parameters:

-n This parameter means --quiet or --silent. Indicates to ignore the output of the execution process and only output our results.
There is another parameter: -i . After using this parameter, all changes will be performed on the original file. Your output will overwrite the original file. Very dangerous, must pay attention.
(2. area:

2,5 means to find the content of lines 2, 3, 4, and 5 in the file.

The specification of this range is very spiritual, such as:

5 Select row 5. 2,5 Select 2 to 5 lines, 4 lines in total. 1~2 selects odd rows, 2~2 selects even rows.

Range selection can also use regular matching. for example:

/void/,+3 Select the line with the word void and the next three lines. 2^void/,/mem/ selects the data between the line starting with void and the line where the word mem appears.

example:

sed -n '5p' sed1.cpp 
sed -n '2,5 p' sed1.cpp 
sed -n '1~2 p' sed1.cpp 
sed -n '2~2 p' sed1.cpp 
sed -n '2,+3p' sed1.cpp 
sed -n '2,$ p' sed1.cpp 
sed -n '/void/,+3 p' sed1.cpp 
sed -n '/^void/,/CLIENT_TYPE_FLAG_BOTH/p' sed1.cpp 
sed -n '/^BroadcastPdu/,/CLIENT_TYPE_FLAG_BOTH/p' sed1.cpp 
sed -n '/^void CImUserManager::BroadcastPdu/,/CLIENT_TYPE_FLAG_BOTH/p' sed1.cpp

(3) Operation:

The most commonly used operation is p, which means print. For example, the following two commands are equivalent:

cat file 
sed -n 'p' file

In addition to printing, there are the following operations:

• p prints the matching content.
• d to delete the matching content. At this time, the -n parameter should be removed. Think about why?
• w Write the matched content elsewhere.
• Although a , i , c and other operations are basic but rarely used, no introduction is given here.

sed -n '2,5 p' sed2.cpp 
sed '2,5 d' sed2.cpp 
sed -n '2,5 w output.txt' sed2.cpp

10.2. Replacement mode

Sed also has a powerful replacement mode, which means to find and replace some of the values, and output the result.

The -n parameter is rarely used with substitution mode.

sed '/^sys/S/a/b/g' filename

There are a lot of parameters in the replacement mode, but both the first part and the fifth part can be omitted. After replacement, the entire text will be output. The first half is used to match some ranges, while the second half performs replacement actions.

(1 Scope:

This range is similar to the range syntax above. for example:

/sys/,+3 Select the line with the word sys and the next three lines. /^sys/,/mem/ Select the line starting with sys and the data between the line with the word mem.

Example:

sed -n '/void/,+3 s/void/int/g' sed2.cpp 
sed '/^void/,/CLIENT_TYPE_FLAG_BOTH/s/ImUser/User/g' sed2.cpp

(2) Command:

The command here refers to s. That is the meaning of substitute. The find part finds the string to be replaced. This part can accept either plain strings or regular expressions. for example:

a Finds the string a in the range line. [a,b,c] Find the string a or b or c from the range line.

Example:

sed 's/a/b/g' file 
sed 's/[a,b,c]/<&>/g' file 

(3) replace:

will replace what is found in the Find Matches section. Unfortunately, regular expressions cannot be used for this part. Commonly used is the exact replacement. For example, replace a with b.

But there are also advanced features. Similar to the regular api of java or python, the replacement of sed also has the meaning of Matched Pattern, and can also get Group, so don't go into it. The commonly used substitute is &, when it is used in the replacement string, it represents the original search matching data.

[&] Indicates that the found data will be surrounded by []. "&" indicates that the searched data is surrounded by "".

The following command will surround each line in the file with quotation marks.

sed 's/.*/"&"/' file 

(4) flag parameters:

These parameters can be used individually or in multiples, only the most commonly used ones are introduced.

• By default, g only matches the first occurrence of the content in the line. Add g to replace the whole text. commonly used.
• p When the -n parameter is used, p will only output the matching line content.
• w Similar to the w mode above, but it only outputs transformed lines.
• The i parameter is more important, indicating that case is ignored. e means that each line that will be output, executes a command. It is not recommended to use xargs to complete this function.

Look at the syntax of the two commands:

sed -n 's/a/b/gipw output.txt' file 
sed 's/^/ls -la/e' file

Eleven, awk column processing

Awk is similar to the sed command, except that sed is good at fetching lines, and the awk command is good at fetching columns. ​

Principle: Generally, it is to traverse each line in a file, and then process each line of the file separately

usage:

awk [可选的命令行选项] 'BEGIN{命令 } pattern{ 命令 } END{ 命令 }'  文件名

11.1. Print certain columns

for example:

$ echo 'I do that' | awk '{print $3 $2 $1}'
thatdoI

Passing the string I do that to the awk command through the pipeline is equivalent to awk processing a file. The content of the file is I do that. By default, spaces are used as separators (no matter how many spaces there are between columns, they will be treated as a space Processing) I do that is divided into three columns. If the separator symbol is ., it can be used like this:

$ echo '192.168.1.1' | awk -F "." '{print $2}'
168

11.2.
The usage of conditional filter awk is as follows:

awk [可选的命令行选项] 'BEGIN{命令 } pattern{ 命令 } END{ 命令 }'  文件名

So how to use the pattern part? Examples are as follows:

$ cat score.txt
tom 60 60 60
kitty 90 95 87
jack 72 84 99
$ awk '$2>=90{print $0}' score.txt
kitty 90 95 87

$2>=90 means that if the value of the second column of the current row is greater than 90, the current row will be processed, otherwise it will not be processed. To put it bluntly, the pattern part is used to filter out the lines that need to be processed from the file for processing, and this part is empty to represent all processing. The pattern part can be the judgment result of any conditional expression, such as >, <, ==, >=, <=, != At the same time, you can also use +, -, *, / to combine compound expressions with conditional expressions , logical &&, ||, ! can also be used. In addition, the pattern part can also use /regular/ to select the lines that need to be processed.

11.3. Judgment statement

The judgment statement is written in the pattern{command} command, which has the same function as conditional filtering, and it can also make the output richer.

$ awk '{if($2>=90 )print $0}' score.txt
kitty 90 95 87
$ awk '{if($2>=90 )print $1,"优秀"; else print $1,"良好"}' score.txt
tom 良好
kitty 优秀
jack 良好
$ awk '{if($2>=90 )print $0,"优秀"; else print $1,"良好"}' score.txt
tom 良好
kitty 90 95 87 优秀
jack 良好

11.4, BEGIN defines header

awk [可选的命令行选项] 'BEGIN{命令 } pattern{ 命令 } END{ 命令 }'  文件名

Example:

$ awk 'BEGIN{print "姓名 语文 数学 英语"}{printf "%-8s%-5d%-5d%-5d\n",$1,$2,$3,$4}' score.txt
姓名 语文数学英语
tom 60 60 60
kitty 90 95 87
jack 72 84 99

It should be noted that in the example, in order to make the output format look good, the operation of left alignment is done (%-8s left alignment, 8 bits wide), and the usage of printf is similar to that of c++. Not only can it be used to define the table header, but also can do some variable initialization work, for example:

$ awk 'BEGIN{OFMT="%.2f";print 1.2567,12E-2}'
1.26 0.12

Here OFMT is a built-in variable that initializes the digital output format and retains two decimal places.

11.5, END add end character

Similar to BEGIN usage

$ echo ok | awk '{print $1}END{print "end"}'
ok
end

11.6. Data calculation

$ awk 'BEGIN{print "姓名 语文 数学 英语 总成绩"; \
sum1=0;sum2=0;sum3=0;sumall=0} \
{printf "%5s%5d%5d%5d%5d\n",$1,$2,$3,$4,$2+$3+$4;\
sum1+=$2;sum2+=$3;sum3+=$4;sumall+=$2+$3+$4}\
END{printf "%5s%5d%5d%5d%5d\n","总成绩",sum1,sum2,sum3,sumall}'\
 score.txt
姓名 语文 数学 英语 总成绩
  tom 60 60 60 180
kitty 90 95 87 272
 jack 72 84 99 255
总成绩 222 239 246 707

Because the command is too long, the \ symbol is used to break the line at the end.

• Output the table header in the BEGIN body, and initialize the four variables to 0.
• Output each line in the pattern body, and accumulate the operation.
• The END body outputs the total statistical results. Of course, a normal person would not input so many formatting symbols to align when using the linux command, so the new command comes column -t again.

11.7. Example of use: network status statistics

Use awk to count some network status of the netstat command, let's take a look at the basic elements of the awk language. The results displayed after the execution of the netstat command indicate the network status of the network connection in column 6. The awk command looks at the statistical results.

netstat  -ant | 
awk ' \
    BEGIN{print  "State","Count" }  \
    /^tcp/ \
    { rt[$6]++ } \
    END{  for(i in rt){print i,rt[i]}  }'
netstat  -ant | 
awk ' \
    BEGIN{print  "State","Count" }  \
    /^tcp/ \
    { if($4=="0.0.0.0:3306" ) rt[$6]++ } \
    END{  for(i in rt){print i,rt[i]}  }'    

Output result:

State Count
LAST_ACK 1
LISTEN 64
CLOSE_WAIT 43
ESTABLISHED 719
SYN_SENT 5
TIME_WAIT 146

Awk is not the same as the usual program, it is divided into four parts:

• BEGIN header, optional. It is used to set some parameters, output some headers, define some variables, etc. The above command only prints one line of information.
• END end section, optional. It is used to calculate some summary logic, or to output these contents. The above command, using a simple for loop, outputs the contents of the array rt.
• The Pattern matching part is still optional. Used to match some rows that need to be processed. The above command only matches the lines beginning with tcp, and the others are not processed.
• Action module. The main logic body, processing by row, and statistical printing are all available.

Notice:

• The main program part of awk is surrounded by single quotes ', not double quotes.
• The starting index of awk column is 0, not 1.
  more details