[Linux study notes 28-3] Shell text processing tool awk

1. Introduction to awk

Awk is a programming language used to process text and data under linux/unix. Data can come from standard input (stdin), one or more files, or the output of other commands. It supports advanced functions such as user-defined functions and dynamic regular expressions, and is a powerful programming tool under linux/unix. It is used on the command line, but more often as a script. Awk has many built-in functions, such as arrays, functions, etc. This is the similarity between it and the C language, flexibility is the biggest advantage of awk

2. awk command

基本结构：
awk 'BEGIN{ print "start" } pattern{ commands } END{ print "end" }' file

1. The BEGIN statement block, the general statement block that can use pattern matching, and the END statement block are composed of three parts,
2. These three parts are optional. Any part can not appear in the script, the script is usually in single or double quotes

/条件/	#条件
/条件1|条件2/	#条件1或者条件2
/条件1/||/条件2/	#条件1或者条件2
/条件1/&&/条件2/	#条件1并且条件2

awk options
-F	Specify input separator
-f	Read awk command from script file

awk built-in variables
NO	Number of lines
NF	Number of columns
FILENAME	file name
“FILENAME”	String
$0	All columns
$1	The first column (the number is the first column)
$2	the second list
^$	Null value

awk operator
||	Logical OR
&&	Logical and
~	Match regular expression
~!	Does not match regular expression
$	Field reference
Space	String concatenation

>>Detailed explanation of regular expressions

3. awk usage example

Show row number, column number

awk '{print NR}' passwd	#显示文件行号
awk '{print $0}' passwd	#显示文件每一列
awk '{print NR" "$0}' passwd	#显示文件行号和内容，中间有空格
awk -F : '{print NF}' passwd	#显示每行的列数（以：为分隔符）

Difference between variable value and string

#无双引号为变量
awk -F : '{print FILENAME}' passwd	#显示文件名称
#有双引号为字符串
awk -F : '{print “FILENAME”}' passwd	#显示字符串

Conditional use example

awk '/bash$|sh$/' /etc/passwd	#显示文件以bash或sh结尾的行
awk -F : '/bash$|sh$/{print $1}' /etc/passwd	#显示指定内容的第一列
awk -F : '/bash$|sh$/&&/^root/{print $1}' /etc/passwd	#以bash或sh结尾的并且以root开头的行的第一列
awk -F : '/bash$|sh$/&&!/^root/{print $1}' /etc/passwd	#以bash或sh结尾的并且不是以root开头的行的第一列

Rows with empty and non-empty values

#第五列值为空的行
awk -F : '$5~/^$/{print}' /etc/passwd
#第五列值不为空的行
awk -F : '$5!~/^$/{print}' /etc/passwd

Awk basic structure example

#执行BEGIN的内容，显示第5列为空的行，执行END内容
awk -F : 'BEGIN{print "hello"}$5~/^$/{print}END{print "111"}' /etc/passwd
#BEGIN定义n，显示第5列为空的行，最后输出n值
awk -F : 'BEGIN{n=0}$5~/^$/{print}END{print n}' /etc/passwd
#BEGIN定义n，每遇到第5列为空的行 执行n++，最后输出n值
awk -F : 'BEGIN{n=0}$5~/^$/{n++}END{print n}' /etc/passwd
#BEGIN定义n，逐行执行n++，最后输出n值
awk -F : 'BEGIN{n=0}{n++}END{print n}' /etc/passwd

4. Testing

Count the number of users who can su-switch in the system and whose home directory is not under /home

#能su-切换的用户
awk -F : '/bash$|sh$/{print}' /etc/passwd
#用户家目录不在/home下的用户
awk -F : '$6!~/^\/home/{print}' /etc/passwd
#能su-切换的并且用户家目录不在/home下的用户
awk -F : '/bash$|sh$/&&$6!~/^\/home/{print}' /etc/passwd
#在系统中能su-切换的并且用户家目录不在/home下的用户数量
awk -F : 'BEGIN{n=0}/bash$|sh$/&&$6!~/^\/home/{n++}END{print n}' /etc/passwd