The significance of the discussion of the command parsing process is as follows:
In the last input and output redirection test program, because of the incomprehension of how the internal analysis is done, the output of the program is in an uncontrollable and incomprehensible state. All is necessary to understand the command line execution process.
First of all, let's talk about the need to know this:
For example: Many examples in https://blog.csdn.net/huayangshiboqi/article/details/80034609 are examples of the parsing process and how to parse the command line.
For example: a=var;b
In this sentence, do you read a=var;b and then start dividing a command, or do you read a=var; first?
Another example: a=1 2 3
How will it be parsed, is it read a=1 2 3 or a=1? After reading the command, how should this sentence be divided? These are questions.
The execution process of the Shell command line is divided into 15 steps:
1. Read the command line
2. Command history replacement
3. Alias substitution
4. Brace expansion
5, tilde replacement
6, I / O heavy redirect
7. Variable substitution
8. Command substitution
9. Word analysis
10. File name generation
11. Quoting character processing
12. Process replacement
13. Environmental treatment
14. Execute the command
15. Track the execution process
The specific implementation of each step is as follows:
1. Read the command line
(1) The first step shell will read the complete command line
Divided into two categories (read command line and read structure statement):
①, read the command line
Shell will judge each character read one by one until it encounters a semicolon ";", background process symbol "&", logical and "&&", logical or "||" or newline characters, etc. The entire command line reading process is over.
②, read a structure statement
When reading a structural statement, such as an if statement block, for statement block, while statement block, etc., the shell will read the entire structure statement. until the above ";", "&", "&&", "||" or newline character is encountered.
Therefore, if there is a redirection > < after the statement block , it will also be regarded as a part and read together.
(2) Syntax analysis of command statements
Action performed:
After reading in a completed command or structural statement, the shell begins to parse the command statement, breaking it down into a series of words or keywords.
Note: Normally, the shell assumes that each "word" or "keyword" is composed of a series of "consecutive characters" separated by spaces or tabs . Even words composed of special characters such as "<" ">" "|" "^".
2. Command history replacement
Analysis: Read the relevant commands from the command history to replace the commands that need to be replaced in the command line.
for example:! ! command, the shell will take the corresponding command from the command history and replace it!!
Note: Command history substitution does not work in non-interactive shells (such as shell scripts). That is, do not use related functions such as history commands in shell scripts.
3. Alias substitution
Analysis: If you have used the aliases command to perform command substitution operations. On the command line, other people's commands will be replaced first.
4. Brace expansion
Analysis: curly braces provide an easy way to generate filenames
For example, the following three commands:
① mkdir ~/scripts/{old,new,tools,admin} #Extension file name
Create four files in the same folder at once
② cp /home/book/src/{main,list.scan.mon}.c #Extension file name
Copy the four files in the /home/book/ directory at one time to the current directory
③ echo char-{one,two,three,four} #Expand string
Display four strings of char-one, char-two, char-three, chat-four at one time
Note: The curly brace expansion mechanism is disabled in non-interactive shells (shell scripts).
5, tilde replacement
The tilde is used to specify the user's home directory.
Question 1: What does the above sentence mean?
·It means that, for example, if you log in to the linux system with the book user, "~" represents /home/book, and if you log in to the linux system with the xiaoming user, "~" represents /home/xiaoming.
No matter which directory you are in, ~ represents the home directory of the currently logged-in active user.
Question 2: How many ways are there to view the user's home directory?
Three viewing methods:
One is: echo $HOME
The second is: echo ~ username For example: echo ~book
The third is: echo ~
Substitution of tildes in commands:
If you have a tilde in your command, the substitution will be done,
① Command: ls –l ~
Indicates to list all files in the user directory
② Command: ls –l ~/music
Indicates to list all files in the music directory under the user's home directory, or list the music files under the home directory
Note: If the ~ is not empty, but is not followed by a valid username or path, the shell will not make any substitutions.
Such as command: echo ~xx
output: ~xx
6, I / O heavy redirect
This step will perform input /output redirection.
Explain with standard output redirection
Action performed:
① Close the specified file descriptor; close file descriptor 1.
② Open the redirected file and save the file pointer information for opening the file
③ Copy the file pointer information to the vacated position in the file pointer data structure table where the original standard output is located.
The process of standard input and standard error output redirection is the same as that of standard output redirection.
7. Variable substitution
Variables involved: internal variables, user-defined variables, references to command-line parameters, references to positional parameters
Meaning: replace these variable references with their original ones ^
8. Command substitution
Command substitution expressions consist of command statements in the form of backticks or $(…). Command substitution is the replacement of the value of a command expression or the command substitution of the command preceding the pipe character.
The command in the command substitution expression can be any Linux command, such as combined command, parallel command of pipe symbol, redirection, etc., without any restrictions.
The command in the command substitution expression is used as a command statement, and any shell mechanism can be used, that is, when the command is executed, it is also executed according to the command line parsing process.
9. Word analysis
Split the command line according to the delimiter to separate the most basic command line elements or words.
Delimiters are typically spaces, tabs, and newlines. Of course, you can also modify the environment variable IFS to increase the needs of some special cases. If there are single and double quotes, the characters inside are taken as a whole, as a word.
IFS setting: IFS="$IFS[followed by a special character you set]"
for example:
At this point, the colon ":" is treated as a delimiter.
However, don't use the delimiter you set yourself directly in the command keyword, such as after echo, as a delimiter, an error will be reported. The delimiter set by yourself is only valid for variables, such as variables read from text, which is the most useful place for the delimiter set by yourself.
10. File name generation
i.e. extended metacharacters. https://blog.csdn.net/huayangshiboqi/article/details/79964711
In this step, each word parsed in the previous step is retrieved one by one to check whether the middle of the word contains any metacharacters that conform to the rules for generating filenames. For example: * ? and so on.
For each base word that contains a metacharacter, the shell will try to match a filename in the current directory or the specified directory in an attempt to expand.
* in double quotes is not interpreted:
https://blog.csdn.net/huayangshiboqi/article/details/80026783
echo * will expand.
11. Quoting character processing
Remove quotation marks, including escape characters, single and double quotation marks and other quotation characters.
In the first ten steps, all expansion, including variable substitution, command substitution, file name generation, etc., has been completed. However, in the first ten steps, some metacharacters in escape characters, single quotes, and double quotes cannot be quoted, and they are retained until step 11. Until the end of the first ten steps, these metacharacters will not be used again. When parsing, put these metacharacters as-is in the middle of the string. (Because the first ten steps are over, you can safely delete single quotes, double quotes, escape characters, etc., because metacharacters will not be parsed again)
Delete escape characters, single and double quotation marks and other quoted characters.
12. Process replacement
Process substitution form: >(command) or <(command)
Process substitution occurs where : where a filename parameter in the command is located
The role of process replacement: to achieve interactive communication between standard input and standard output between processes
Process replacement mechanism: There are two special files in the /dev/fd directory as intermediate files for inter-process communication. For example, command1 >(command2), it writes the standard output of command1 to a file in the /dev/fd directory, and then redirects the file to command2 as standard input.
Example:
① Compare the differences between two files
diff <(ls -l testfile1) <(ls -l testfile2)
Note that the process substitution form appears where the filename parameter should appear
② Sort
sort -n -k 5 <(ls -l)
③ A wrong usage
ls -l> (grip test)
Usage error.
The original intention is to use the standard output of ls -l as the standard input of the grep command, and grab the part containing the string test in the middle.
However, the position of the process substitution form is used incorrectly, and the process substitution form can only appear in the position where the filename parameter is located.
This can be changed to a pipe character, or grep test <(ls -l)
One thing to note here is that although the process replacement is executed later, it does not mean that the parsing starts after all the previous steps have been executed.
For example, if you think that process substitution is performed after removing the double quotes, then process substitution can be used casually in double quotes, which is obviously wrong.
Script example test.sh:
a=test
grep "${a}" <(echo /home/book/test*)
Debug run: bash -x test.sh
Output information:
+ a=test
+ grep test / dev / fd / 63
++ echo /home/book/test /home/book/test.sh
/home/book/test /home/book/test.sh
Analysis: The second sentence in the script is divided into two commands to execute.
First is grep test /dev/fd/63. where test is the replacement of "${a}" and /dev/fd/63 is a special file during process replacement.
It can be seen from this step that the variable substitution has been completed and the double quotes have been removed, but the process substitution has not been performed.
Then the execution process replaces echo /home/book/test /home/book/test.sh, and then writes the output to /dev/fd/63,
Finally, execute the grep command.
There is a very important thing in the middle: /dev/fd/63.
So, since the double quotes are removed before the process replacement, can the process replacement be written inside the double quotes?
The answer is: no!
Because before the double quotes are parsed, process substitution will have a symbolic file /dev/fd/63. If the process substitution is written within double quotes, the process substitution is treated as a normal string, and no intermediate file /dev/fd/63 is created.
Therefore, in the next step, this string of ordinary strings will not be executed as process replacement.
13. Environmental treatment
It's actually a search command.
Retrieve the storage location of the command file according to the path in the environment variable PATH, and then replace the command name in the command line with the full path name of the command file.
14. Execute the command
There are four types of commands to be executed: Linux commands, internal shell commands, applications, and shell scripts.
① If it is an ordinary Linux command, the Shell will start a separate subprocess to execute the corresponding command;
② If it is an internal command of the shell, it will be executed directly by the shell.
③ If it is an application, the shell subprocess will call the exec system call to execute the application.
④ If it is a shell script, each line of statements in the file will be executed by the shell interpreter.
How to distinguish between applications and shell scripts:
The child process will try to load the command file into memory and then start executing. If it cannot be loaded into the process, the Forbidden City determines that the corresponding command file is a shell script, so the shell executes the file by means of interpretation.
15. Track the execution process
debugging
If you use options such as -v or -x of the set command (commands for debugging) in the command, the execution of the command is traced. The shell will output the currently executing statement.
The characteristics of the command statement output at this time:
① The command actually executed. is the command actually executed after command substitution
② Output in sequence according to the logical order of the commands