Shell programming regular expression-awk tool use

1. The regular expression of Shell programming-use of awk tool

1. Built-in variables of awk

Awk contains several special built-in variables (which can be used directly) as follows:

• FS: Specify the field separator for each line of text, the default is a space or tab stop.
• NF: The number of fields in the currently processed row.
• NR: The row number (ordinal number) of the row currently being processed.
• $0: The entire line of the currently processed line.
• $n: The nth field (nth column) of the currently processed row.
• FILENAME: The name of the file being processed.
• RS: Data record separation, the default is \n, that is, one record per line.

2. Find out the username, user ID, and group ID of /etc/passwd

awk -F: ‘{print $1,$3,$4}’ /etc/passwd

3. Output all content, which is equivalent to cat test.txt

awk ‘{print}’ test.txt

4. Output all content, which is equivalent to cat test.txt

awk ‘{print $0}’ test.txt

5. Output the content of lines 1~3

awk ‘NR==1,NR==3{print}’ test.txt

6. Output the content of lines 1~3

awk ‘(NR>=1)&&(NR<=3){print}’ test.txt

7. Output the contents of the first and third lines

awk ‘NR==1||NR==3{print}’ test.txt

8. Output the contents of all odd lines

awk ‘NR%2==1{print}’ test.txt

9. Output the contents of all even lines

awk ‘NR%2==0{print}’ test.txt

10. Output the line starting with root

awk ‘/^root/{print}’ /etc/passwd

11. Output the line ending with nologin

awk ‘/nologin$/{print}’ /etc/passwd

12. Count the number of lines ending in /bin/bash

Same as grep -c “/bin/bash$” /etc/passwd

awk ‘BEGIN {x=0};//bin/bash$/{x++};END {print x}’ /etc/passwd

13. Count the number of text paragraphs separated by blank lines

awk ‘BEGIN {RS=""};END{print NR}’ test.txt

14. Output the third field in each line (separated by spaces or tab stops)

awk ‘{print $3}’ test.txt

15. Output the first and third fields in each row

awk ‘{print $1,$3}’ test.txt

16. Output the shadow record of the user whose password is empty

awk -F: ‘$2==""{print}’ /etc/shadow

17. Output the shadow record of the user whose password is empty

awk ‘BEGIN{FS=":"};$2==""{print}’ /etc/shadow

18. The output is separated by a colon and the first field of the line containing /bash in the seventh field

awk -F: ‘$7~"/bash"{print $1}’ /etc/passwd

19. Call the wc -l command to count the number of users using bash

Same as grep -c "bash$" /etc/passwd

awk -F: ‘/bash$/{print| “wc -l”}’ /etc/passwd

20. Invoke the w command and use it to count the number of online users

awk ‘BEGIN {while (“w” | getline) n++;{print n-2}}’

21. Call hostname and output the current hostname

awk ‘BEGIN {“hostname” | getline;print $0}’

22. Common options for the sort tool

• -f: Ignore case;
• -b: Ignore the spaces in front of each line;
• -M: sort by month;
• -n: sort by number;
• -r: reverse sorting;
• -u: equivalent to uniq, which means that only one line of the same data is displayed;
• -t: Specify the separator, use [Tab] to separate by default;
• -o <output file>: dump the sorted results to the specified file;
• -k: Specify the sorting area.

23. Sort the accounts in the /etc/passwd file

sort /etc/passwd

24. Sort the third column in the /etc/passwd file in reverse order

sort -t ‘:’ -rk 3 /etc/passwd

25. Sort the third column in the /etc/passwd file, and save the output to the user.txt file

sort -t ‘:’ -k 3 /etc/passwd -o user.txt

cat user.txt

26. Common options for uniq tools

• -c: count;
• -d: Only display duplicate lines;
• -u: Only display lines that appear once.

27, delete duplicate lines in the file

uniq testfile

28. Delete duplicate lines in the file, and display the number of repeated occurrences of the line at the beginning of the line

uniq -c testfile

29, find duplicate lines in the file

uniq -d testfile

30. Common options for tr tools

• -c: Replace all characters that do not belong to the first character set;
• -d: delete all characters belonging to the first character set;
• -s: Represent the continuously repeated characters as a single character;
• -t: Delete characters in the first character set more than the second character set.

31. Convert input characters from uppercase to lowercase

echo “KGC” | tr ‘A-Z’ ‘a-z’

32. Compress repeated characters in input

echo “thissssss is a test linnnnnnnnne.” | tr -s ‘sn’

33, delete some characters in the string

echo “hello world” | tr -d 'from'