Collect 20+ command line artifacts that improve development efficiency N times

Recommended reading:

• I summarized 72 interview questions, totaling 3170 pages, and won 30+ Internet company offers (including BATJM)

• I won the first battle in 2020. This Java Interview Magic Plus version allowed me to successfully get offers from major manufacturers such as Ali, JD.com, and Bytedance.

• Worship! The K8S (kubernetes) study guide that Ali is pushing internally, can’t be more detailed

background

This article is mainly derived from a small sharing within the group of the previous company, organized into an article po. The title is "Shell Boosts Development Efficiency", and the more relevant one is "Command Line" to improve development efficiency. Shell programming is not mentioned here, but mainly introduces some basic tool commands commonly used under Linux or Mac to help deal with some Daily affairs.

Through the introduction of this article, you should have a preliminary understanding of the relevant commands, and know for example what commands can be used to accomplish what operations. As for the specific parameters, you do not need to memorize them deliberately. When you need them, go  cmd --help or  man cmduse If more, the commonly used commands will naturally be remembered.

This article first introduces some commonly used command line tools under Linux/Mac, then uses specific examples to illustrate the commonly used command usage, and finally uses one or two cases to illustrate the power of these tools:

• For example, given an nginx log file, what is the top 10 that can find the most HTTP 404 requests? For example, what is the top 10 that can find the most time-consuming requests?
• For another example, what is the hourly "PV" that can be easily obtained? Another example is to get an article. Can you simply count the 10 words with the highest single-time word frequency in this article?
• You need to change the file names in a certain folder in batches, compress the pictures in the folder to a fixed size in batches, and so on.

Mac environment

• zsh

• on-my-zsh

• plugin

• git

• car dump

• osx(man-preview/quick-look/pfd(print Finder director)/cdf(cd Finder))

• Commonly used shortcut keys ( bindkey)

• Demo: highlight/git/smart completion/jump (j, d)...

Shell basic commands

• which / whereis, common  whatis,  man, --help

➜  .oh-my-zsh git:(master)$ whereis ls
/bin/ls
➜  .oh-my-zsh git:(master)$ which ls
ls: aliased to ls -G

• Basic file directory operations

rm， mkdir， mv， cp， cd， ls， ln， file， stat， wc(-l/w/c)， head， more， tail， cat...

• Sharp weapon pipeline: |

Shell text processing

Here is a case study of the general usage and parameters of the 12 commands. You can click on the directory on the right (my blog has a directory, and the official account is on the wood) to go directly to the command you want to know.

find, grep, xargs, cut, paste, comm
join, sort, uniq, tr, sed, awk

find

• Common parameters

• File name  -name, file type -type, search depth-maxdepth

• Time filtering (create/access/modify) -[cam]time

• Perform actions -exec

• Example

find ./ -name "*.json"
find . -maxdepth 7 -name "*.json" -type f
find . -name "*.log.gz" -ctime +7 -size +1M -delete (atime/ctime/mtime)
find . -name "*.scala" -atime -7 -exec du -h {} \;

grep

• Common parameters

• -v(invert-match)，

• -c(count)，

• -n(line-number)，

• -i(ignore-case)，

• -l， -L， -R(-r， --recursive)， -e

• Example

grep 'partner' ./*.scala -l
grep -e 'World' -e 'first' -i -R ./  (-e: or)

• Related commands: grep -z / zgrep / zcat xx | grep

xargs

• Common parameters

• -n (number of columns per row),

• -I (variable substitution)

• -d (separator), Mac does not support, pay attention to the difference with the GNU version

• Example

echo "helloworldhellp" | cut -c1-10
cut -d， -f2-8 csu.db.export.csv

cut

• Common parameters

• -b (byte)

• -c (character)

• -f (column), -d (separator), f range: n, n-, -m, n-m

• Example

echo "helloworldhellp" | cut -c1-10cut -d， -f2-8 csu.db.export.csv

paste

• Common parameters

• -d separator

• -s column to row

• Example

    ➜  Documents$ cat file1
1 11
2 22
3 33
4 44
➜  Documents$ cat file2
one     1
two     2
three   3
one1    4

➜  Documents$ paste -d, file1 file2
1 11, one     1
2 22, two     2
3 33, three   3
4 44, one1    4
➜  Documents$ paste -s -d: file1 file2
a 11:b bb:3 33:4 44
one     1:two     2:three   3:one1    4

join

Similar in sql  ...inner join ...on ...,  -t separator, the default is a space or tab

➜  Documents$ cat j1
1 11
2 22
3 33
4 44
5 55
➜  Documents$ cat j2
one     1   0
one     2   1
two     4   2
three   5   3
one1    5   4
➜  Documents$ join -1 1 -2 3 j1 j2
1 11 one 2
2 22 two 4
3 33 three 5
4 44 one1 5

comm

• Common parameters

• usage comm [-123i] file1 file2

• Dictionary sequence , 3 columns: only in file1/file2/both

• - Remove a column, i ignore case

• Example

    ➜  Documents$ seq 1 5 >file11
➜  Documents$ seq 2 6 >file22
➜  Documents$ cat file11
1
2
3
4
5
➜  Documents$ cat file22
2
3
4
5
6
➜  Documents$ comm file11 file22
1
        2
        3
        4
        5
    6
➜  Documents$ comm -1 file11 file22
    2
    3
    4
    5
6
➜  Documents$ comm -2 file11 file22
1
    2
    3
    4
    5
➜  Documents$ comm -23 file11 file22
1

Related commands  diff (similar to git diff )

sort

• Common parameters

• -d， --dictionary-order

• -n， --numeric-sort

• -r， --reverse

• -b， --ignore-leading-blanks

• -k， --key

• Example

➜  Documents$ cat file2
one     1
two     2
three   3
one1    4
➜  Documents$ sort file2
one     1
one1    4
three   3
two     2
➜  Documents$ sort -b -k2 -r file2
one1    4
three   3
two     2
one     1

uniq

• Common parameters

• -c number of repetitions

• -d duplicate

• -u not repeated

• -f ignore the first few columns

• Example

➜  Documents$ cat file4
11
22
33
11
11
➜  Documents$ sort file4 | uniq -c
   3 11
   1 22
   1 33
➜  Documents$ sort file4 | uniq -d
11
➜  Documents$ sort file4 | uniq -u
22
33
➜  Documents$ cat file3
one     1
two     1
three   3
one1    4
➜  Documents$ uniq -c -f 1 file3
   2 one     1
   1 three   3
   1 one1    4

Note: uniqcompare whether adjacent ones are repeated, generally used in sortconjunction with

tr

• Common parameters

• -c complement

• -d delete

• -s compress adjacent duplicates

• Example

➜  Documents$ echo '1111234444533hello' | tr  '[1-3]' '[a-c]'
aaaabc44445cchello
➜  Documents$ echo '1111234444533hello' | tr -d '[1-3]'
44445hello
➜  Documents$ echo '1111234444533hello' | tr -dc '[1-3]'
11112333
➜  Documents$ echo '1111234444533hello' | tr -s '[0-9]'
123453hello
➜  Documents$ echo 'helloworld' | tr '[:lower:]' '[:upper:]'
HELLOWORLD

and

• Common parameters

• -d delete

• -s replacement, g global

• -e multiple commands overlay

• -i modify the original file (add parameter "" under Mac, backup)

• Example

    ➜  Documents$ cat file2
one     1
two     2
three   3
one1    4
➜  Documents$ sed "2,3d" file2
one     1
one1    4
➜  Documents$ sed '/one/d' file2
two     2
three   3
➜  Documents$ sed 's/one/111/g' file2
111     1
two     2
three   3
1111    4
#将one替换成111 并将含有two的行删除
➜  Documents$ sed -e 's/one/111/g' -e '/two/d' file2
111     1
three   3
1111    4
# ()标记(转义), \1 引用
➜  Documents$ sed 's/\([0-9]\)/\1.html/g' file2
one     1.html
two     2.html
three   3.html
one1.html    4.html
# 与上面一样 & 标记匹配的字符
➜  Documents$ sed 's/[0-9]/&.html/g' file2
one     1.html
two     2.html
three   3.html
one1.html    4.html
➜  Documents$ cat mobile.csv
"13090246026"
"18020278026"
"18520261021"
"13110221022"
➜  Documents$ sed 's/\([0-9]\{3\}\)[0-9]\{4\}/\1xxxx/g' mobile.csv
"130xxxx6026"
"180xxxx8026"
"185xxxx1021"
"131xxxx1022"

awk

• Basic parameters and syntax

• NR row number, NF column number

• $1 Column 1,  $2, $3…

• -F fs fs delimiter, string or regular

• Syntax:, the  awk 'BEGIN{ commands } pattern{ commands } END{ commands }'process is as follows:

1. Execute begin
2. Execute for each line of input  pattern{ commands }, pattern can be regular /reg exp/, relational operation, etc.
3. After processing, execute end

• Example

➜  Documents$ cat file5
11  11 aa cc
22  22 bb
33  33 d
11  11
11  11
#行号， 列数量， 第3列
➜  Documents$ awk '{print NR"("NF"):"， $3}' file5
1(4): aa
2(3): bb
3(3): d
4(2):
5(2):
#字符串分割， 打印1，2列
➜  Documents$ awk -F"xxxx" '{print $1， $2}' mobile.csv
"130 6026"
"180 8026"
"185 1021"
"131 1022"
#添加表达式
➜  Documents$ awk '$1>=22 {print NR":"， $3}' file5
2: bb
3: d
#累加1到36，奇数，偶数
➜  Documents$ seq 36 | awk 'BEGIN{sum=0; print "question:"} {print $1" +"; sum+=$1} END{print "="; print sum}' | xargs | sed 's/+ =/=/'
question: 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18 + 19 + 20 + 21 + 22 + 23 + 24 + 25 + 26 + 27 + 28 + 29 + 30 + 31 + 32 + 33 + 34 + 35 + 36 = 666
➜  Documents$ seq 36 | awk 'BEGIN{sum=0; print "question:"} $1 % 2 ==1 {print $1" +"; sum+=$1} END{print "="; print sum}' | xargs | sed 's/+ =/=/'
question: 1 + 3 + 5 + 7 + 9 + 11 + 13 + 15 + 17 + 19 + 21 + 23 + 25 + 27 + 29 + 31 + 33 + 35 = 324
➜  Documents$ seq 36 | awk 'BEGIN{sum=0; print "question:"} $1 % 2 !=1 {print $1" +"; sum+=$1} END{print "="; print sum}' | xargs | sed 's/+ =/=/'
question: 2 + 4 + 6 + 8 + 10 + 12 + 14 + 16 + 18 + 20 + 22 + 24 + 26 + 28 + 30 + 32 + 34 + 36 = 342

Other high-level syntax: for, while etc., various functions, etc., awkis a powerful language in itself , and you can master some basic usage.

Practical application

Log statistics analysis

For example, when you get an nginx log file, you can do many things, such as looking at which requests are the longest time-consuming and then optimizing, such as looking at the number of "PV" per hour and so on.

➜  Documents$ head -n5 std.nginx.log
106.38.187.225 - - [20/Feb/2017:03:31:01 +0800] www.tanglei.name "GET /baike/208344.html HTTP/1.0" 301 486 "-" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 360JK yunjiankong 975382" "106.38.187.225, 106.38.187.225" - 0.000
106.38.187.225 - - [20/Feb/2017:03:31:02 +0800] www.tanglei.name "GET /baike/208344.html HTTP/1.0" 301 486 "-" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 360JK yunjiankong 975382" "106.38.187.225, 106.38.187.225" - 0.000
10.130.64.143 - - [20/Feb/2017:03:31:02 +0800] stdbaike.bdp.cc "POST /baike/wp-cron.php?doing_wp_cron=1487532662.2058920860290527343750 HTTP/1.1" 200 182 "-" "WordPress/4.5.6; http://www.tanglei.name/baike" "10.130.64.143" 0.205 0.205
10.130.64.143 - - [20/Feb/2017:03:31:02 +0800] www.tanglei.name "GET /external/api/login-status HTTP/1.0" 200 478 "-" "-" "10.130.64.143" 0.003 0.004
10.130.64.143 - - [20/Feb/2017:03:31:02 +0800] www.tanglei.name "GET /content_util/authorcontents?count=5&offset=0&israndom=1&author=9 HTTP/1.0" 200 11972 "-" "-" "10.130.64.143" 0.013 0.013

The above is a case of nginx, for example, you want to find the path requested by the top 10:

head -n 10000 std.nginx.log | awk '{print $8 ", " $10}' | grep ',404' | sort | uniq -c | sort -nr -k1 | head -n 10
#or
head -n 10000 std.nginx.log | awk '$10==404 {print $8}' |sort | uniq -c | sort -nr -k1 | head -n 10

Of course, you may not directly process it successfully at one time. Generally, you will first take part of the data for processing to see if the logic is normal, or you can cache some intermediate results.

cat std.nginx.log | awk '{print $8 "," $10}' | grep ',404' >404.log
sort 404.log | uniq -c | sort -nr -k1 | head -n 10

For example, the number of requests per hour, request time, etc.

➜  Documents$ head -n 100000 std.nginx.log | awk -F: '{print $1 $2}' | cut -f3 -d/ | uniq -c
8237 201703
15051 201704
16083 201705
18561 201706
22723 201707
19345 201708

Other practical cases ip block

Case: db data correction

Background: Because a service bug, leading to insert into db image path does not need to be of the form (security needs have sensitive data replacement)  https://www.tanglei.name/upload/photos/129630//internal-public/shangtongdai/2017-02-19-abcdefg-eb85-4c24-883e-hijklmn.jpg replaced  http://www.tanglei.me/internal-public/shangtongdai/2017-02-19-abcdefg-eb85-4c24-883e-hijklmn.jpgas mysql and other db seemingly does not support direct regular replacement, it can not be easily Write SQL for replacement (even if you support it, it is risky to change it directly, you should back it up and then modify it to leave a "regret medicine").

Of course, exporting the data and then writing python and other script processing is also a solution, but if you use the above command line to process it, it only takes tens of seconds to complete.

step:

1. Prepare data

select id, photo_url_1, photo_url_2, photo_url_3 from somedb.sometable where 
photo_url_1 like 'https://www.tanglei.name/upload/photos/%//internal-public/%' or
photo_url_2 like 'https://www.tanglei.name/upload/photos/%//internal-public/%' or
photo_url_3 like 'https://www.tanglei.name/upload/photos/%//internal-public/%';

2. When replacing the original file with sed, first test whether it is replaced normally.

#测试是否OK
head -n 5 customers.csv | sed 's|https://www.tanglei.name/upload/photos/[0-9]\{1,\}/|http://www.tanglei.me|g'
# 直接替换原文件， 可以sed -i ".bak" 替换时保留原始备份文件
sed -i "" 's|https://www.tanglei.name/upload/photos/[0-9]\{1,\}/|http://www.tanglei.me|g' customers.csv

3. Splice sql, and then execute

awk -F， '{print "update sometable set photo_url_1 = " $2, ", photo_url_2 = " $3, ", photo_url_3 = " $4, " where id = " $1 ";" }' customers.csv > customer.sql
#然后执行sql 即可

other

• play framework session

The old way: need to start the play environment, slow. The new way is solved directly by the command line.

sbt "project site" consoleQuick
import play.api.libs._
val sec = "secret...secret"
var uid = "10086"
Crypto.sign(s"uid=$uid"， sec.getBytes("UTF-8")) + s"-uid=$uid"

➜  Documents$  ~/stdcookie.sh 97522
918xxxxdf64abcfcxxxxc465xx7554dxxxx21e-uid=97522
➜  Documents$ cat ~/stdcookie.sh
#!/bin/bash ##  cannot remove this line
uid=$1
hash=`echo -n "uid=$uid" | openssl dgst -sha1 -hmac "secret...secret"`
echo "$hash-uid=$uid"

• Counting word frequency in articles: The following case counts the 10 most frequent words in the original text of Trump's inauguration speech.

➜  Documents$ head -n3 chuanpu.txt
Chief Justice Roberts， President Carter， President Clinton， President Bush， President Obama， fellow Americans and people of the world， thank you.

We， the citizens of America， are now joined in a great national effort to rebuild our country and restore its promise for all of our people. Together we will determine the course of America and the world for many， many years to come.
➜  Documents$ cat chuanpu.txt | tr -dc 'a-zA-Z ' | xargs -n 1 | sort | uniq -c | sort -nr -k1 | head -n 20
  65 the
  63 and
  48 of
  46 our
  42 will
  37 to
  21 We
  20 is
  18 we
  17 America
  15 a
  14 all
  13 in
  13 for
  13 be
  13 are
  10 your
  10 not
  10 And
  10 American

• Random number: For example, often register a new website and randomly generate a password.

➜  Documents$ cat /dev/urandom | LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 5
cpBnvC0niwTybSSJhUUiZwIz6ykJxBvu
VDP56NlHnugAt2yDySAB9HU2Nd0LlYCW
0WEDzpjPop32T5STvR6K6SfZMyT6KvAI
a9xBwBat7tJVaad279fOPdA9fEuDEqUd
hTLrOiTH5FNP2nU3uflsjPUXJmfleI5c
➜  Documents$ cat /dev/urandom | head -c32 | base64
WoCqUye9mSXI/WhHODHDjzLaSb09xrOtbrJagG7Kfqc=

• Image processing compression, batch size change, etc. sips

➜  linux-shell-more-effiency$ sips -g all which-whereis.png
/Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
  pixelWidth: 280
  pixelHeight: 81
  typeIdentifier: public.png
  format: png
  formatOptions: default
  dpiWidth: 72.000
  dpiHeight: 72.000
  samplesPerPixel: 4
  bitsPerSample: 8
  hasAlpha: yes
  space: RGB
  profile: DELL U2412M
➜  linux-shell-more-effiency$ sips -Z 250 which-whereis.png
/Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
  /Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
➜  linux-shell-more-effiency$ sips -g all which-whereis.png
/Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
  pixelWidth: 250
  pixelHeight: 72
  typeIdentifier: public.png
  format: png
  formatOptions: default
  dpiWidth: 72.000
  dpiHeight: 72.000
  samplesPerPixel: 4
  bitsPerSample: 8
  hasAlpha: yes
  space: RGB
  profile: DELL U2412M
➜  linux-shell-more-effiency$ sips -z 100 30 which-whereis.png
/Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
  /Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
➜  linux-shell-more-effiency$ sips -g pixelWidth -g pixelHeight which-whereis.png
/Users/tanglei/Documents/linux-shell-more-effiency/which-whereis.png
  pixelWidth: 30
  pixelHeight: 100

• Command-line processing JSON artifact: With the versatility of JSON, it is often necessary to process JSON data. This command-line JSON processing artifact is recommended here. jq is a lightweight and flexible command-line JSON processor[1]