Chapter 1 Shell Overview

Why should big data programmers learn Shell?

Need to understand the Shell program written by operation and maintenance personnel.
Occasionally, I write some simple Shell programs to manage the cluster and improve development efficiency.

shell script?
Before talking about what a shell script is, let's talk about what a shell is.
Shell means shell, which is the shell of the operating system. We can operate and control the operating system through shell commands. For example, Shell commands in Linux include ls, cd, pwd, and so on. In summary, Shell is a command interpreter, which starts, pauses, and stops the running of programs or controls the computer by accepting Shell commands entered by the user.

Shell is an application program that connects users to the Linux kernel, allowing users to use the Linux kernel more efficiently, securely, and at low cost. This is the essence of Shell.

The shell itself is not part of the kernel, it is just an application written on the basis of the kernel.
Insert picture description here
So what is a shell script?

A shell script is an execution file composed of Shell commands, which integrates some commands into a file to process business logic. The scriptRun without compiling。It interprets and runs through an interpreter, so the speed is relatively slow。

Chapter 2 Shell Parser

The Shell parser provided by Linux is: cat /etc/shells
Insert picture description here
The relationship between bash and sh: ll | grep bash The

default parser of Centos is bash: echo $SHELL

Chapter 3 Getting Started with Shell Scripts

1. Script format The
script starts with ==#!/bin/bash== (specify parser)
2．The first Shell script: helloworld

Requirements: Create a Shell script and output helloworld

Common execution methods of scripts(Commonly used ./script name, sh script name, bash script name)

Increase execution authority +x: chmod 777 script name
(Using sh, bash execution, the essence is that the bash parser helps you execute the script, so the script itself does not need execution permission. Without sh, bash execution, the essence is that the script needs to be executed by itself, so the execution permission is required)

3. The second Shell script: multi-command processing
Insert picture description here

Chapter 4 Variables in Shell

4.1 System variables

Insert picture description here

4.2 Custom variables (variable=value, unset variable)

Declare static variables: readonly variables, note: cannot unset

Variable definition rules

The variable name can be determined byLetters, numbers and underscoresComposition, butCannot start with a number, Environment variable names are recommended to be capitalized.
There can be no spaces on both sides of the equal sign
In bash, the default types of variables are string types,Cannot perform numerical calculations directly。
If the value of the variable has spaces, it needs to be enclosed in double or single quotation marks。

Insert picture description here

4.3 Special variables: $n $# $* $@ $?

Special variable: $n

$n	（功能描述：n为数字，$0代表该脚本名称，$1-$9代表第一到第九个参数，十以上的参数，十以上的参数需要用大括号包含，如${10}）

Special variable: $#

1. Basic grammar

$#	（功能描述：获取所有输入参数个数，常用于循环）。

Output the script file name, input parameter 1 and input parameter 2 values, and count the number of input parameters
Insert picture description here

Special variables : $* $@

$*	（功能描述：这个变量代表命令行中所有的参数，$*把所有的参数看成一个整体）
$@	（功能描述：这个变量也代表命令行中所有的参数，不过$@把每个参数区分对待）

Insert picture description here

（2）比较$*和$@区别
（a）$*和$@都表示传递给函数或脚本的所有参数，不被双引号“”包含时，都以$1 $2 …$n的形式输出所有参数。
（b）当它们被双引号“”包含时，“$*”会将所有的参数作为一个整体，以“$1 $2 …$n”的形式输出所有参数；“$@”会将各个参数分开，以“$1” “$2”…”$n”的形式输出所有参数。

[atguigu@hadoop101 datas]$ vim for.sh

#!/bin/bash 

for i in "$*" 
#$*中的所有参数看成是一个整体，所以这个for循环只会循环一次 
        do 
                echo "ban zhang love $i"
        done 

for j in "$@" 
#$@中的每个参数都看成是独立的，所以“$@”中有几个参数，就会循环几次 
        do 
                echo "ban zhang love $j" 
done

[atguigu@hadoop101 datas]$ chmod 777 for.sh
[atguigu@hadoop101 datas]$ bash for.sh cls xz bd
ban zhang love cls xz bd
ban zhang love cls
ban zhang love xz
ban zhang love bd

Special variables: $?

$？	（功能描述：最后一次执行的命令的返回状态。如果这个变量的值为0，证明上一个命令正确执行；如果这个变量的值为非0（具体是哪个数，由命令自己来决定），则证明上一个命令执行不正确了。）

Chapter 5 Operators

1．基本语法
（1）“$((运算式))”或“$[运算式]”
（2）expr  + , - , \*,  /,  %    加，减，乘，除，取余

注意：expr运算符间要有空格

Insert picture description here
Calculate the value of (2+3) X4

using the $[Expression] method

Chapter 6 Conditional Judgment

1. Basic grammar
[condition] (note that there must be spaces before and after the condition)
Note: If the condition is not empty, it is true, [atguigu] returns true, and [] returns false. ,

（1）两个整数之间比较
= 字符串比较
-lt 小于（less than）			-le 小于等于（less equal）
-eq 等于（equal）				-gt 大于（greater than）
-ge 大于等于（greater equal）	-ne 不等于（Not equal）

（2）按照文件权限进行判断
-r 有读的权限（read）			-w 有写的权限（write）
-x 有执行的权限（execute）

（3）按照文件类型进行判断
-f 文件存在并且是一个常规的文件（file）
-e 文件存在（existence）		-d 文件存在并是一个目录（directory）

(1) Whether 23 is greater than or equal to 22
Insert picture description here
(2) Whether helloworld.sh has write permission

[atguigu@hadoop101 datas]$ [ -w helloworld.sh ]
[atguigu@hadoop101 datas]$ echo $?
0

(3) Does the file in the /home/atguigu/cls.txt directory exist?

[atguigu@hadoop101 datas]$ [ -e /home/atguigu/cls.txt ]
[atguigu@hadoop101 datas]$ echo $?
1

(4) Multi-condition judgment (&& means that the next command is executed only when the previous command is executed successfully, || means that the next command is executed only after the previous command has failed to execute)
Insert picture description here

Chapter 7 Process Control (Key Points)

7.1 if judgment

if [ 条件判断式 ];then 
  程序 
fi 

或者 
if [ 条件判断式 ] 
  then 
    程序 
fi

注意事项：
（1）[ 条件判断式 ]，中括号和条件判断式之间必须有空格
（2）if后要有空格

(1) Input a number, if it is 1, output banzhang zhen shuai, if it is 2, output cls zhen mei, if it is other, output nothing.
Insert picture description here

7.2 case statement

case $变量名 in 
"值1"） 
    如果变量的值等于值1，则执行程序1 
;; 
"值2"） 
    如果变量的值等于值2，则执行程序2 
;; 
  …省略其他分支… 
*） 
    如果变量的值都不是以上的值，则执行此程序 
;; 
esac


1：双分号“;;”表示命令序列结束，相当于java中的break。
2：最后的“*）”表示默认模式，相当于java中的default。

(1) Input a number, if it is 1, then output..., if it is 2, then output..., if it is other, output....
Insert picture description here

7.3 for loop

for (( 初始值;循环控制条件;变量变化 )) 
  do 
    程序 
  done

(1) Increase from 1 to 100

Insert picture description here

for 变量 in 值1 值2 值3… 
  do 
    程序 
  done

(1) Print all input parameters
Insert picture description here

7.4 while loop

while [ 条件判断式 ] 
  do 
    程序
  done

(1)
Insert picture description here
Improve from 1 to 100

Chapter 8 read read console input

read(选项)(参数)
	
选项：
 -p：指定读取值时的提示符；
 -t：指定读取值时等待的时间（秒）。
参数
	变量：指定读取值的变量名

(1) Within 7 seconds of the prompt, read the name entered on the console
Insert picture description here

Chapter 9 Functions (basename, dirname, custom functions)

1. Basic syntax of basename

basename [string / pathname] [suffix]  	（功能描述：basename命令会删掉所有的前缀包括最后一个（‘/’）字符，然后将字符串显示出来。
选项：
suffix为后缀，如果suffix被指定了，basename会将pathname或string中的suffix去掉。

(1) Intercept the file name of the /root/datas/banzhang.txt path

Insert picture description here
2. Basic syntax of dirname

dirname 文件绝对路径		（功能描述：从给定的包含绝对路径的文件名中去除文件名（非目录的部分），然后返回剩下的路径（目录的部分））

(1) Path to obtain banzhang.txt file
Insert picture description here
3: Custom function

[ function ] funname[()]
{
    
    
	Action;
	[return int;]
}
funname

(1) Calculate the sum of two input parameters

Insert picture description here

Chapter 10 Shell Tools (Key Points)

10.1 cut

Insert picture description here
(1) Select the system PATH variable value, all the paths after the second ":" starts:

(2) The IP address printed after cutting ifconfig

but 10.2

Insert picture description here
(1) Insert the word "United States" under the second line of sed.txt, and print

(2) Delete all lines containing ang in

the sed.txt file (3) Replace ang in the sed.txt file with waves

Note:'g' means global, replace all

10.3 awk

Insert picture description here
(0) Data preparation

(1) Search for all lines in the passwd file starting with the root keyword, and output the 7th column of the line.

(2) Search for all lines of the passwd file starting with the root keyword, and output the first and seventh columns of the line, separated by "," in the middle.
Insert picture description here
(3) Only display the first and seventh columns of /etc/passwd, separated by commas, and add the column name user in front of all rows, and the shell adds "dahaige,/bin/zuishuai" in the last row.

Note: BEGIN is executed before all data read lines; END is executed after all data is executed.

(4) Increase the user id in the passwd file by 1 and output
Insert picture description here
the built-in variables of awk

Insert picture description here
(1) Count the file name of passwd, the line number of each line, the number of columns in each line

(2) Cutting IP

(3) Query the line number of the blank line in sed.txt