Shell scripts in-depth tutorial (1): Getting Started

Shell Script Basics

Bash Comment

Bash supports only single-line comments, use #the beginning of the statement are interpreted as comments:

# 整行注释
echo hello world # 行尾注释

Bash through some of the features can be tricky to achieve multi-line comments:

: '
注释1
注释2
'

: <<'EOF'
注释1
注释2
EOF

____='
注释1
注释2
'

However, do not pain free eggs tricky to use multi-line comments, peace of mind #to comment.

Bash basic data types

Only basic data types Bash string type, numeric type are not even ( declare -ito force values declared type).

such as:

# 都会当作字符串
echo haha
echo 1234

Bash string concatenation

Bash operating in series strings directly connected to the two pieces of data, without any operator.

E.g:

echo "junma""jinlong"
echo 1234 5678

Command Basics

Variable assignment and reference variable

a=3
echo $a

a="www.junmajinlong.com"
echo $a

a='hello world'
echo $a

Shell can reference an undefined variable:

echo $xyzdefabc

Empty variables can be defined:

a=
echo $a

Variable Substitution

Variable substitution means at a position before the start of execution, the value of the variable Shell will replace the first reference variable in the command.

E.g:

a="hello"
echo $a world

Before the echo command begins execution, Shell will get a variable value hello, and replace it with the command line $aat. So, when the echo command execution, command line has become:

echo hello world

In addition to variable substitution, Shell will make other alternatives:

• Command substitution
  
• Process substitution
  
• Replace Arithmetic
  
• Brace expansion
  
• Tilde expansion
  
• Path extension

These extensions and replacements are Shell command before the call is complete, this and other language parsing code in a different way.

The back will explain in detail how Shell is doing command-line parsing, if you do not master the command line parsing, when faced with the command-line syntax error is likely to spend a lot of unnecessary time to debug commands. And the mastery of command line parsing, command will be in charge of the life cycle of such, I can not say one will be able to write all the command line, but it can save a lot of debugging time, to write the command line and the ability to write the script will rise a level .

Command substitution

Backticks or $()can execute command substitution.

`cmd`
$(cmd)

First substitution means performs command cmd, cmd and outputs the result to replace $()or at a position back quotes.

E.g:

echo `id root`
echo $(id root)

Before the echo command is executed, the first execution will result id command, the command id:

$ id root
uid=0(root) gid=0(root) groups=0(root)

So the result will be uid=0(root) gid=0(root) groups=0(root)replaced $(id root). So, when the echo command execution, command line has become:

echo uid=0(root) gid=0(root) groups=0(root)

Arithmetic

$[]Or $(())or let command can do the arithmetic.

let alone command, the other can not write the command line.

a=3
let a=a+1
echo $a

$[]And $(())can be written on the command line inside, Shell when parsing the command line, they will do the arithmetic, and then replace the computation result to the command line.

a=33
echo $[a+3]
echo $((a+3))

Because variable substitution to replace in arithmetic, therefore, use a variable name or reference variable methods can be:

a=333
echo $[$a+3]
echo $(($a+3))

Exit status codes

After each command execution there will be a corresponding process exit status codes that indicate whether or not the process is normal exit.

Therefore, the command line, Shell script, often using special variables $?determine whether a recent reception command normal exit.

Typically, if $?the value of:

• 0 indicates successful execution of the process, that normal exit
  
• Non-zero, it indicates that the process is not successfully executed, that is, non-normal exit
  
• But non-zero exit status does not necessarily indicate an error, it may be normal logical exit

In addition, Shell script, all conditional (such as an if statement, while statement) expressed True 0 exit status to a non-zero exit status to False.

exit command

exit command can be used to exit the current Shell process, such as Shell exits the current terminal, exit Shell scripts, and so on.

exit [N]

exit exit status codes may specify N, omitted, if N, the default exit status is 0, it means that the correct exit.

Background execution Command &

At the end use the command &symbol, this command can be placed in the background.

After the command into the background, the process will immediately return to Shell, Shell process will execute immediately the next command (if any) or to exit.

Use $!can get a PID recent background process.

sleep 20 &
echo $!

Use waitthe command can wait for a background process (the current child process Shell process) is completed:

wait [n1 n2 n3 ...]

When not given any parameters to, we will wait for all child processes (that is, all background process) is completed.

sleep 5 &
wait
echo haha

Multi-command combination

Shell There are several ways of combining a plurality of commands.

After 1.cmd1 quit, do cmd2

cmd1;cmd2

After 2.cmd1 correct exit (exit status 0), execution cmd2

cmd1 && cmd2

After 3.cmd1 does not exit properly, perform cmd2

cmd1 || cmd2

4. logics: &&and ||may be combined arbitrarily

# cmd1正确退出后执行cmd2，cmd2正确退出后执行cmd3
cmd1 && cmd2 && cmd3...

# cmd1正确退出则执行cmd2，cmd1不正确退出会执行cmd3
# cmd1正确退出，但cmd2不正确退出，也会执行cmd3
cmd1 && cmd2 || cmd3

# cmd1正确退出会执行cmd3
# cmd1不正确退出会执行cmd2，cmd2正确退出会执行cmd3
cmd1 || cmd2 && cmd3

5. The plurality of command packet: parentheses or brackets plurality of commands may be combined

# 小括号组合的多个命令是在子Shell中执行
# 即会先创建一个新的Shell进程，在执行里面的命令
(cmd1;cmd2;cmd3)

# 大括号组合的多个命令是在当前Shell中执行
# 大括号语法特殊，要求：
#   1.开闭括号旁边都有空白，否则语法解析错误(解析成大括号扩展)
#   2.写在同一行时，每个cmd后都要加分号结尾
#   3.多个命令可分行书写，不要求分号结尾
{ cmd1;cmd2;cmd3; }
{
  cmd1
  cmd2
  cmd3
}

Basic redirection

Software Design believes that the program should have a data source, data export and reporting the wrong place. In the Linux system, each program will be open three default file descriptor (file descriptor, fd):

• fd = 0: standard input, indicates where the data read from the default program
  
• fd = 1: standard output, the program indicates where the data is output to the default
  
• fd = 2: standard error, indicates that the program will output an error message to default where

File descriptor, it means the file system to open track allocated to it a number, file number and the corresponding relationship: reading data from a file descriptor, it means that the corresponding data is read from a file, the file write data descriptor, it means that data is written to the corresponding file.

Linux, everything is a file, the file descriptor is a file. default:

• fd = 0 is a standard input / dev / stdin file
  
• fd = 1 standard output is / dev / stdout file
  
• fd = standard error 2 is / dev / stderr file

The default file is a soft link to the file each terminal:

$ ls -l /dev/std*
lrwxrwxrwx 1 root root 15 Jan  8 20:26 /dev/stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Jan  8 20:26 /dev/stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Jan  8 20:26 /dev/stdout -> /proc/self/fd/1

$ ls -l /proc/self/fd/
lrwx------ 1 root root 64 Jan 16 10:40 0 -> /dev/pts/0
lrwx------ 1 root root 64 Jan 16 10:40 1 -> /dev/pts/0
lrwx------ 1 root root 64 Jan 16 10:40 2 -> /dev/pts/0
lr-x------ 1 root root 64 Jan 16 10:40 3 -> /proc/75220/fd

Therefore, by default all read and write data terminals, for example:

# 数据输出到终端
$ echo haha
$ cat /etc/fstab

# 从终端读取数据
$ cat
hello   # 在终端输入
hello   # 在终端输出
world   # 在终端输入
world   # 在终端输出
^C

Changing the target file descriptor corresponding to the data flow can be changed. Fd = 1 such as a standard default input flow is a terminal device, which if changed /tmp/a.log, it allows the file data is written /tmp/a.log rather than the terminal device.

In the Shell, this change of behavior is called the target file descriptor to redirect that redefine the flow of data.

In fact, there are many types of file descriptor operations, including the redirection fd, fd is assigned (Open, i.e., open the file), fd copy (Duplicate), fd movement (Move), fd is closed (close). Now only describes the basic redirection.

Shell, based redirection in the following ways:

• [n]>file: Overlay output redirection, fd = n output to change to the output data file to file, create file does not exist, the presence of the file data is written again to empty
  • When omitted, n- >file, equivalent to 1>file, i.e., the standard output is redirected to the file file cover
    
• [n]>>file: Append Output redirect output to fd = n changes to the output data file to the end of the file, the file does not exist to create, append file is present directly in the end of the file
  • When omitted, n- >>file, equivalent to 1>>file, i.e. to the standard output file file append redirection
    
• [n]<file: Input redirection, open the file in read mode, and the file allocation fd = n, file does not exist error
  • When omitted, n- <file, equivalent to 0<file, i.e. read data directly from file
    
  • Generally only the program data read from the fd = 0, so that when n is not equal to 0, the extra step required operation 3<file <&3, read skip
    
• &>file: This is a special redirection methods, represent the standard error and standard output is redirected to a file file, equivalent to>file 2>&1
  
• &>>file: This is a special redirection methods, represent the standard error and standard output file are appended to the document, is equivalent to>>file 2>&1

Furthermore, a special object file is often used for the output / dev / null, which device is empty, it may lose all write data directly.

echo www.junmajinlong.com >/dev/null
curl -I www.junmajinlong.com 2>/dev/null >/tmp/a.log

cat </etc/fstab

A frequently used technique is the way to empty file:

$ cat /dev/null >file
$ >file

Distinguish cat <file cat file and

cat is a command, the command's source code to write some code to handle options and parameters.

cat -n /etc/fstab

After the cat command execution, identifies -noptions, while the output line numbers, while the cat will recognize this option will make cat output /etc/fstabparameters, cat reads the parameters specified file and output.

If the file does not specify the parameters of the cat, the cat will default to read data from standard input. The default standard input is a terminal, so there is no change in the flow of standard input, reads data from the terminal, which is what the user input characters, reads what character, what character then output:

$ cat
junmajinlong   # 在终端输入
junmajinlong   # 在终端输出
junma   # 在终端输入
junma   # 在终端输出
^C

But the user can change the standard input sources. such as:

$ cat </etc/fstab

It represents the standard input source to the / etc / fstab file, then reads data from the cat / etc / fstab in.

Moreover, convention, would use a -to represent standard input or standard output.

# 下面是等价的，都表示从标准输入中读取数据
cat -
cat /dev/stdin
cat

Note: This is not infallible, but most of the code in the program of the convention defines -the relevant code processing. Refer to the relevant command man page. As man cat in a line:

With no FILE, or when FILE is -, read standard input.

here doc

Input Redirection <, in addition <<、<<<.

<<Notation here doc. In other words, it is followed with a document, like a file, but this is a temporary file is defined in <<the symbol. here doc used to specify multiple data input.

Since it is a document, there are documents from the notation from the beginning of the document and the document terminating symbol indicates the end of this document. Start and end in the middle of all the contents of that document content. Content of the document will be read as a data standard input.

Document start and end characters can be arbitrarily defined, but must be the same both before and after. Common symbols are:

• EOF：end of file
• EOL：end of line
• EOB：end of block

E.g:

# here doc作为标准输入被读取，然后被cat输出
cat <<EOF
hello
world
EOF

# here doc的内容还会被cat覆盖式输出到指定文件
cat <<eof >/tmp/file
hello
world
eof

# here doc的内容还会被cat追加式输出到指定文件
cat <<eof >>/tmp/file
hello
world
eof

# here doc和重定向符号的前后位置随意
cat >>/tmp/file<<eof
...
eof

Further, if the start code value is quoted, is not variable substitution, command substitution, replacement, etc. arithmetic. If not quoted start character, it will be replaced.

a=333
cat <<eof
$a
eof

cat <<"eof"
$a
eof

Output:

333
$a

here string

<<<He expressed here string. That is behind the symbol is a string that will be entered as content standards.

cat <<<"www.junmajinlong.com"

When using single quotes here string, not variable substitution, replacement, etc. command, use can be replaced when surrounded by double quotes.

$ a=3333
$ cat <<<$a            
3333
$ cat <<<"hello world$a"
hello world3333
$ cat <<<'hello world$a' 
hello world$a

here string can often replace the pipe before the echo command echo xxx|. E.g:

# 下面是等价的
echo hello world | grep "llo"
grep "llo" <<<"hello world"

pipeline

Pipeline usage:

cmd1 | cmd2 | cmd3...

Each vertical bar represents a pipeline. The above command will be represented cmd1 standard output into the duct, cmd2 reads from the pipeline processing, cmd2 standard output will be placed in another conduit, cmd3 reads data from the pipeline processing. Back can also take any number of pipes.

Shell Shell pipeline is one of the most commendable features, it implements inter-process communication pipes in a very simple form, I think half of Shell processing text data comes from the form of vertical pipes. Like other programming languages, open the pipeline should write to distinguish which process pipeline, which process reading the pipe, for safety, but also to shut down each process not read or write terminal end, in short, it is in trouble, and Shell pipeline is very simple, vertical line on the left is to write pipeline, the right of the vertical line is read pipe.

E.g:

ps aux | grep 'sshd'

Data (Standard Output) PS command generation will be written into the pipe, as long as there is data in the pipeline, grep command to read data for processing.

That the following command, grep reads data from which it?

ps aux | grep '#' /etc/fstab

Grep both want to make it read data from / etc / fstab, read the data from the pipe species it?

ps aux | grep '#' /etc/fstab /etc/stdin
ps aux | grep '#' /etc/fstab -

The tee command

tee command can be copied as a standard input and standard output to 0 or more files. In other words, tee role of multiple data redirection.

NAME
    tee - read from standard input and write to standard output and files

SYNOPSIS
    tee [OPTION]... [FILE]...

DESCRIPTION
    Copy standard input to each FILE, and also to standard output.

    -a, --append
        ppend to the given FILEs, do not overwrite

Figure:

E.g:

$ echo hello world | tee /tmp/file1 /tmp/file2 | cat
$ echo hello world | tee -a /tmp/file3 >/dev/null

Process substitution

Bash also supports the process of replacing (Note: Some Shell does not support the process of replacement).

Process substitution syntax:

<(cmd)
>(cmd)

Similar processes and command substitution, let cmd command is executed first, because they are in the Shell command line parsing stage of implementation.

Process substitution is performed asynchronously let cmd into the background and does not wait for cmd executed.

In fact, each process is replaced by a virtual file, but this file is (produced by cmd command <(cmd)) or cmd command read ( >(cmd)).

$ echo <(echo www.junmajinlong.com)
/dev/fd/63

Since the process of replacing the file, then it can be operated like a file. Such as read, it is redirected as a standard input data sources, and so:

# cmd做数据产生者
$ cat <(echo www.junmajinlong.com)   # 等价于cat /dev/fd/63
$ cat < <(echo www.junmajinlong.com) # 等价于cat </dev/fd/63

# cmd做数据接收者
$ echo hello world > >(grep 'llo')
$ echo hello world | tee >(grep 'llo') >(grep 'rld') >/dev/null

Test conditions statement

test command or functionally equivalent Bash built-in commands [ ]can do a conditional test, if the test result is True, the exit status is 0.

In addition, it can also be used [[]]to do the testing conditions, even let、$[]、$(())can do a conditional test, but it would not introduce here.

These conditions are commonly used in the tests if, while statement, also commonly used in cmd1 && cmd2 || cmd3command line format.

Example usage:

sh_file=test.sh
[ -x "$sh_file" ] && ./$sh_file || { echo "can't execute,exit...";exit 1; }
test -x "$sh_file" && ./$sh_file || { echo "can't execute,exit...";exit 1; }

[]The test conditions required for opening and closing brackets separated by spaces, otherwise a parser error (parsed into wildcard).

No test content

[  ]
test

The absence of any test content, direct returns false.

true and false command

Returns true true commands directly, i.e., as exit status 0.

false commands directly returns false, i.e., non-zero exit status.

$ true
$ echo $?  # 0
$ false
$ echo $?  # 1

File type test

Conditional expression	meaning
-e file	If a file exists (exist)
-f file	If a file exists and is a regular file (file)
-d file	If a file exists and is a directory (directory)
-b file	If the file exists and block device to a block device
-c file	If a file exists and is a character device character device
-S file	If the file exists and is a socket file Socket
-p file	If a file exists and is a named pipe file FIFO (pipe)
-L file	If the file exists and is a file link (Link)

File attribute class test

Conditional expression	meaning
-r file	If the file exists and the current user-readable
-w file	If the file exists and the current user can write
-x file	If the file exists and the current user can perform
-s file	If the file size is greater than 0 and bytes exists, i.e., whether the document is detected non-empty file
-N file	If the file exists, and since it was last read whether modify

Comparison between the two documents

Conditional expression	meaning
file1 file2 -nt	If (newer than) than file1 file2 new judge
file1 file2 -ot	If (older than) judgment file1 is older than file2
-If file1 file2	(Equal file) to determine whether file1 file2 with the same file

Numerical comparison size

Conditional expression	meaning
int1 -eq int2	Two values ​​are equal (equal)
int1 -ne int2	Two numerical ranges (not equal)
int1 -gt int2	n1 is greater than n2 (greater than)
int1 -lt int2	n1 is smaller than n2 (less than)
int1 -ge int2	greater than or equal n1 n2 (greater than or equal)
-The int1 int2	less n1 n2 (less than or equal)

String comparison

Conditional expression	meaning
-with page	(Zero) to determine whether the string is empty? str is the empty string, the true
Street -n str	Determine whether the string is not empty? str is a string, then false. NOTE: -n may be omitted
str1 = str2 str1 == str2	str1 and str2 are the same, the same is true returns. "==" and "=" Equivalent
str1 != str2	str1 is not equal to str2, if not, etc., then return true
str1 > str2	Alphabetical str1 is greater than str2, returns true if more than
str1 < str2	Alphabetical str1 is less than str2, returns true if less than

Logical Operators

Conditional expression	meaning
-a或&&	(and) two expressions at the same time when true is true. "-a" or can only be used in the test [], the && only in [[]] in
-o or ||	(or) a true expression of any two was true. "-o" or can only be used in the test in [], only in || [[]] in
!	The expression negated
( )	Change the priority of the expression, in order to prevent the shell being parsed, be escaped with a backslash ()

The if statement

if test-commands; then
  consequent-commands;
[elif more-test-commands; then
  more-consequents;]
[else alternate-consequents;]
fi

test-commands may be either the test or test []、[[]]to test, but may be any other command, the test conditions for test-commands, command only determines whether the exit status of 0, 0 is then true.

E.g:

if [ "$a" ];then echo '$a' is not none;else echo '$a' undefined or empty;fi

if [ ! -d ~/.ssh ];then
  mkdir ~/.ssh
  chown -R $USER.$USER ~/.ssh
  chmod 700 ~/.ssh
fi

if grep 'junmajinlong' /etc/passwd &>/dev/null;then
  echo 'User "junmajinlong" already exists...'
elif grep 'malongshuai' /etc/passwd &>/dev/null;then
  echo 'User "malongshuai" already exists...'
else
  echo 'you should create user,exit...'
  exit 1
fi

case

commonly used in the case of determining the branch determination. such as:

case "$1" in
    start)
        echo start;;
    stop)
        echo stop
        ;;
    restart)
        echo restart
        ;;
    reload | force-reload)
        echo reload;;
    status)
        echo status;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload}"
        exit 2
esac

case use basic requirements:

• In addition to the last branch, each branch to ;;the end, otherwise appears branch penetration (it is ;;not necessary)
  
• Branch condition can use wildcard
  
• The branch condition may be used a plurality of spaced apart vertical condition, if it matches one of which represents the implementation of the branch
  
• Finally, we define a generally matches any other default branch condition, i.e.,*)

for loop

There are two structures for loop:

# 成员测试类语法
for i in word1 word2 ...;do cmd_list;done

# C语言for语法
for (( expr1;expr2;expr3 ));do cmd_list;done

Members of the class for the test cycle, the key is used in one or more elements separated by spaces, when the for loop, each taken from a key element in the rear and assigned to the variable i.

E.g:

$ for i in 1 2 3 4;do echo $i;done
1
2
3
4
$ for i in 1 2 "3 4";do echo $i;done
1
2
3 4

C language type for grammar, expr1 is an initialization statement, expr2 is the end of the loop conditional statement, expr3 statement is executed after each cycle, typically used to change the criteria to determine the relevant variables.

for ((i=1;i<=3;++i));do echo $i;done
1
2
3

while loop

while test_cmd_list;do cmd_list;done

while loop, will be at the start of the test test_cmd_list, if the test exit status is zero, then perform a loop statement cmd_list, then test test_cmd_list, has been circulating until the test is non-zero exit status, loop exits.

E.g:

let i=1,sum=0;
while [ $i -le 10 ];do 
  let sum=sum+i
  let ++i
done

There until loop, but rarely used in the Shell.

The while loop is often used commands and read together, read a Bash built-in commands can be used to read the file, usually by the line that reads: each read one line.

E.g:

cat /etc/fstab | while read line;do
  let num+=1
  echo $num: $line
done

The above command line, the first cat and process structure began while running, while the structure of the read command reads from standard input, data is read from the pipe, each read a line, because there is no data pipeline initially, so read command data is blocked in a wait state. When the cat command file read all the data, the data is placed into the pipe, and then the read command reads one line read from the pipe lines each assigned to a variable Line, and execute the body, and the cycle continues until the read After reading all of the data, the loop exits.

But note that both sides of the pipeline is executed command defaults in the sub-Shell, so variable set disappeared after the command execution is complete. In other words, the parent Shell can not access these variables. For example, the above num variable is set in while the structure of the pipeline, in addition to the access while in the variable anywhere else can not access it.

If you want to visit while in the variable assignment, can not use the pipeline. If the file is read directly from, you can use the input redirection, if a read command is the data generated using alternative processes.

while read line;do
  let num1+=1
  echo $num1: $line
done </etc/fstab
echo $num1

while read line;do
  let num2+=1
  echo $num2: $line
done < <(grep 'UUID' /etc/fstab)