Work Act terminal 10X (a)

table of Contents

1. Terminal
2. Grep
3. however,
4. Awk
5. Xargs
6. Find

In the above github have a star of more than 700 repo is called Bash-Oneliner, introduced many practical and can effectively improve the efficiency of command, we have to find out. Original Direct: Bash-oneliner . Note: The removed portion of the command looks futile, you can see all the original content.

1. Terminal

Note: Here is the terminal behavior under Linux, Mac will be slightly different in the following, try to completion

Ctrl-related

Ctrl + n : 类似向下方向键
Ctrl + p : 类似向上方向键
Ctrl + r : 反向搜索 terminal 的历史命令
Ctrl + s : 停止该 terminal
Ctrl + q : 在 Ctrl + s 后面重新恢复该 terminal
Ctrl + a : 移动光标到行的开始处。(这个很有用)
Ctrl + e : 移动光标到行的结尾处。(同上)
Ctrl + d : 如果当前的 terminal 的命令行有输入，那么 Ctrl + d 会删除光标处的数字。否则会退出当前的 terminal
Ctrl + k : 删除从当前光标开始到结尾的所有的字符
Ctrl + x + backspace: 删除从当前光标到行开始的所有的字符
Ctrl + t : 交换当前光标下的字符和其前面的字符的位置。Esc + t 交换光标前面的两个单词。(这个很有意思)
Ctrl + w : 剪切光标之前的单词。Ctrl + y 粘贴该单词
Ctrl + u : 剪切光标之前的所有字符。Ctrl + y 粘贴刚刚剪切的字符
Ctrl + _ : 撤销前面的操作。(可以连续操作多次)
Ctrl + l : 类似 clear。(类似 Mac 终端下的 Command + k)
Ctrl + x + Ctrl + e : 唤起 $EDITOR 环境变量设置的编辑器程序，在需要输入多行的情况下比较有用。(试验了一下如果之前没有设置，$EDITOR 的默认值是 emacs。你可以将其设置为 vim，即 export EDITOR=vim。当然也可以将这个环境变量设置为其他的应用程序去实现一些有趣的功能，这个测试过了)

Change the character case

Esc + u : 将当前光标开始到单词结尾的字符都转换成大写。(这里的单词指的是光标所在的位置前后以空格分隔形成的单词)
Esc + l : 将当前光标开始到单词结尾的字符都转换成小写
Esc + c : 将光标所在位置的字符转换成大写

Command execution history

!53 : 执行 history 中的 53 号命令
!! : 执行上一条命令

A command to replace, and replace some of the parameters

# 上一条命令：echo 'aaa'
^aaa^bbb
# 此时将上一条命令替换为 echo 'bbb'，输出 bbb

# 需要注意的是，这样只会替换第一次 aaa，如果要替换所有的 aaa，需要像下面这样使用
^aaa^bbb^:&
# 或者
!!:gs/aaa/bbb/

Prefix match command execution history

!cat
# 或者
!c
# 执行历史命令中最近一条满足前缀是 c 或者 cat 的命令

File name using regular

# ? 表示一个单独的任意字符
/b?n/?at  # 匹配上 /bin/cat

# * 表示多个任意字符
/etc/pa*wd  # 匹配上 /etc/passwd

# '[]' 表示一个字符范围
ls -al [a-z]*  # 列出所有以字母开头的文件

# '{}' 文件名匹配多种模式
ls {*.sh,*.py}  # 列出所有的 .sh 和 .py 文件

Environment Variables

$0, $1, $2, $3, ... ： 在执行 shell 的时候传参使用，$0 表示 shell 名字，$1,$2,$3依次表示后面的参数
$# : 参数的个数
$? : 最近的一个终端 foreground 命令的退出状态
$- : 当前 shell 设置的选项，可以通过 echo $- 查看
$$ : 当前 shell 进程的 pid
$! : 最近的一个终端后台命令的 pid

$DESKTOP_SESSION : 当前的展示管理器。(可能说的是 xWindow)
$EDITOR : 编辑器，可以通过上面提到的快捷键唤醒
$LANG : 语言设置
$PATH : 这个不用说了
$PWD : 当前目录
$SHELL : 当前的 shell
$USER : 当前的 username
$HOSTNAME : 当前的 hostname

2. Grep

Kind of grep

grep = grep -G  # 支持基本的正则表达式
fgrep = grep -F # 查找文件里符合条件的字符串
egrep = grep -E # 支持扩展的正则表达式
pgrep = grep -P # 兼容 Perl 的正则表达式语法
rgrep = grep -r # 递归 grep

Count the number of empty lines

grep -c "^$"

grep and returns only numbers

grep -o '[0-9]*'
#或者
grep -oP '\d'

grep containing specific digital number

grep ‘[0-9]\{3\}’
# or
grep -E ‘[0-9]{3}’
# or
grep -P ‘\d{3}’

Find the IP address

grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'

Find the word

# return also 3 lines after match
grep -A 3 'bbo'

# return also 3 lines before match
grep -B 3 'bbo'

# return also 3 lines before and after match
grep -C 3 'bbo'

Find the beginning of the specific character of the word

grep -o 'S.*'

Extract text between two particular words

grep -o -P '(?<=w1).*(?=w2)'

Find the file content does not contain a word

grep -v bbo filename

Find content file is not a specific character at the beginning of

grep -v '^#' file.txt

Find what with spaces

bbo="some strings"
grep "$boo" filename

Find the first match content files

grep -m 1 bbo filename

Finds and returns to meet the conditions of the contents of the file number of

grep -c bbo filename

Count the number of times the word appears in the document

grep -o bbo filename |wc -l

Find case sensitive

grep -i "bbo" filename

Coloring match result

grep --color bbo filename

Find all files in a directory

grep -R bbo /path/to/directory
# or
grep -r bbo /path/to/directory

Find all files in a directory, not the content of the output file

grep -rh bbo /path/to/directory

Find all files in the directory, only the output matching file names

grep -rl bbo /path/to/directory

OR Find

grep 'A\|B\|C\|D'

AND search (such as A and B)

grep 'A.*B'

Regular find (such as ACB, or AEB)

grep 'A.B'

Find specific characters (such as color or colour)

grep ‘colou?r’

Find all the contents of multiple files

grep -f fileA fileB

Find tab

grep $'\t'

Find variables

$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
#grep -q will output 0 if match found
#remember to add space between []!

Find the middle of the string brackets

grep -oP '\(\K[^\)]+'

Skip directory lookup

grep -d skip 'bbo' /path/to/files/*

3. however,

The first line is removed

sed 1d filename

100 lines before removing files

sed 1,100d filename

Remove the file line contains specific strings

Note: This method does not modify the original file, you can redirect the output to a new file is saved

sed "/bbo/d" filename
# case insensitive:
sed "/bbo/Id" filename

To remove a file does not meet the n-th string is not equal to a value of row

# 第 5 个字符不等于 2
sed -E '/^.{5}[^2]/d'
#aaaa2aaa (you can stay)
#aaaa1aaa (delete!)

Modify the original file

# 删除包含 bbo 的行并直接保存文件
sed -i "/bbo/d" filename

Use variables when using double quotes

# e.g. add >$i to the first line (to make a bioinformatics FASTA file)
sed "1i >$i"
# notice the double quotes! in other examples, you can use a single quote, but here, no way!
# '1i' means insert to first line

Delete blank lines

sed '/^\s*$/d'

# or

sed '/^$/d'

Delete the last line

sed '$d'

The last character deleted files

sed -i '$ s/.$//' filename

Insertion string to the beginning of the file (such as "[")

sed -i '1s/^/[/' file

Insertion string to a file in a particular row

sed -e '1isomething' -e '3isomething'

Insert characters to the end of the file (such as "]")

sed '$s/$/]/' filename

Insert a new row to the end of the file

sed '$a\'

Insert data to each line of the file

sed -e 's/^/bbo/' file

Insert data to the end of each line of the file

sed -e 's/$/\}\]/' filename

Newline character is inserted every n (for example 4 characters each)

sed 's/.\{4\}/&\n/g'

Connect multiple files

sed -s '$a,' *.json > all.json

Replaces

sed 's/A/B/g' filename

Replace the contents of the file based on the regular

sed "s/aaa=.*/aaa=\/my\/new\/path/g"

Filter file to a particular string starting line

sed -n '/^@S/p'

Multi-line print file

sed -n 500,5000p filename

In particular the print file line

sed -n '0~3p' filename

# 打印 3 的倍数行

Print odd lines

sed -n '1~2p' filename

Delete the beginning of the file space and tab

sed -e 's/^[ \t]*//'
# Notice a whitespace before '\t'!!

Delete only spaces

sed 's/ *//'

# notice a whitespace before '*'!!

Removing the end of the file comma

sed 's/,$//g'

End of file add an (tab-delimited)

sed "s/$/\t$i/"
# $i is the valuable you want to add

# To add the filename to every last column of the file
for i in $(ls);do sed -i "s/$/\t$i/" $i;done

Print specific rows

sed -n -e '123p'

The last character deleted files

sed '$ s/.$//'

Specify the position of the insert character

sed -r -e 's/^.{3}/&#/' file

4. Awk

Set tab as the delimiter

awk -F $'\t'

Tab to set output content delimiter

awk -v OFS='\t'

Passing parameters

a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename

The number of characters in the output file and line number of each line

awk '{print NR,length($0);}' filename

The number of output column / field of

awk '{print NF}'

Judgment is not a comma

awk '$1~/,/ {print}'

All the output lines before the string appears n times

awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'

The output file name and its last line

ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file

Inserting the string into the designated column

awk 'BEGIN{OFS="\t"}$3="chr"$3'

Line contains a specific character string is removed

awk '!/bbo/' file

Removes the last column

awk 'NF{NF-=1};1' file

Understand the role of FNR and NR

# For example there are two files:
# fileA:
# a
# b
# c
# fileB:
# d
# e
awk 'print FILENAME, NR,FNR,$0}' fileA fileB
# fileA    1    1    a
# fileA    2    2    b
# fileA    3    3    c
# fileB    4    1    d
# fileB    5    2    e

5. Xargs

Set tab as the delimiter

xargs -d\t

Each row displays three elements

echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3
# 4 5 6

Before performing the first inquiry

$ echo a b c |xargs -p -n 3
$ echo a b c ?... #输入 y
$ a b c

find and delete files

find . -name "*.html"|xargs rm

Delete the file name of the file contains spaces

find . -name "*.c" -print0|xargs -0 rm -rf

Display limits

xargs --show-limits

Moving Files

find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old

# or
find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old

ls |head -100|xargs -I {} mv {} d1

Concurrent execution

time echo {1..5} |xargs -n 1 -P 5 sleep

# a lot faster than:
time echo {1..5} |xargs -n1 sleep

Based on the condition of the file copy

find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B

# v: verbose|
# p: keep detail (e.g. owner)

And with the use of sed

ls |xargs -n1 -I file sed -i '/^Pos/d' filename

The file name is added to the first line of the file

ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt

statistics

ls |xargs -n1 wc -l

Will be integrated into the output line

ls -l| xargs

Statistics folder each file the following number of lines and the total number of rows

ls|xargs wc -l

And joint use grep

cat grep_list |xargs -I{} grep {} filename

And joint use sed

grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'

6. Find

Recursively list all the files and subdirectories

find .

List all files in the current directory

find . -type f

Lists all subdirectories in the current directory

find . -type d

Modify all files in the current directory (the 'www' replaced 'w')

find . -name '*.php' -exec sed -i 's/www/w/g' {} \;

# if there are no subdirectory
replace "www" "w" -- *
# a space before *

Delete the file size is less than 74 byte

find . -name "*.mso" -size -74c -delete

# M for MB, etc