HDFS study notes (2): HDFS common operation commands (Hadoop version 3.3.1)

command format

HDFS command format 1: hadoop fs -cmd <args>
HDFS command format 2:hdfs dfs -cmd <args>

-help

• Use format: -help [command option]
• Meaning analysis: help
• Command example:

[hadoop@node3 testData]$ hdfs dfs -help ls
-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...] :
  List the contents that match the specified file pattern. If path is not
  specified, the contents of /user/<currentUser> will be listed. For a directory a
  list of its direct children is returned (unless -d option is specified).
  
  Directory entries are of the form:
  	permissions - userId groupId sizeOfDirectory(in bytes)
  modificationDate(yyyy-MM-dd HH:mm) directoryName
  
  and file entries are of the form:
  	permissions numberOfReplicas userId groupId sizeOfFile(in bytes)
  modificationDate(yyyy-MM-dd HH:mm) fileName
  
    -C  Display the paths of files and directories only.
    -d  Directories are listed as plain files.
    -h  Formats the sizes of files in a human-readable fashion
        rather than a number of bytes.
    -q  Print ? instead of non-printable characters.
    -R  Recursively list the contents of directories.
    -t  Sort files by modification time (most recent first).
    -S  Sort files by size.
    -r  Reverse the order of the sort.
    -u  Use time of last access instead of modification for
        display and sorting.
    -e  Display the erasure coding policy of files and directories.

-ls

• Use format: -ls <path>
• Meaning Analysis: View the directory structure of the specified path
• Command example:

[hadoop@node3 ~]$ hdfs dfs -ls /
Found 3 items
drwxr-xr-x   - hadoop supergroup          0 2022-06-20 14:38 /hbase
drwxrwx---   - hadoop supergroup          0 2022-07-21 15:05 /tmp
drwxr-xr-x   - hadoop supergroup          0 2022-07-21 09:08 /user

-lsr

• Use format: -lsr <path> or -lsR <path>
• Meaning analysis: recursively view the directory structure of the specified path
• Command example:

[hadoop@node3 ~]$ hdfs dfs -lsr /user/hadoop
lsr: DEPRECATED: Please use 'ls -R' instead.
drwxr-xr-x   - hadoop supergroup          0 2022-07-21 15:10 /user/hadoop/wordcount
drwxr-xr-x   - hadoop supergroup          0 2022-07-21 17:08 /user/hadoop/wordcount/input
-rw-r--r--   3 hadoop supergroup         52 2022-07-21 17:26 /user/hadoop/wordcount/input/echo_test.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-21 16:27 /user/hadoop/wordcount/input/word_count_test.txt
drwxr-xr-x   - hadoop supergroup          0 2022-07-21 15:10 /user/hadoop/wordcount/output
-rw-r--r--   3 hadoop supergroup          0 2022-07-21 15:10 /user/hadoop/wordcount/output/_SUCCESS
-rw-r--r--   3 hadoop supergroup         54 2022-07-21 15:10 /user/hadoop/wordcount/output/part-r-00000

-of

• Use format: -du <path>
• Meaning analysis: the size of each file in the statistical directory
• Command example:

[hadoop@node3 ~]$ hdfs dfs -du /
59126   177378   /hbase
850533  2551599  /tmp
175     525      /user

-So

• Use format: -du <path> or -du -s <path>
• Meaning analysis: the size of the entire file (folder) in the statistical directory
• Command example:

[hadoop@node3 ~]$ hdfs dfs -dus /
dus: DEPRECATED: Please use 'du -s' instead.
909834  2729502  /

-count

• Use format: -count [-q] <path>
• Meaning analysis: count the number of files (folders) in the directory
• Command example:

[hadoop@node3 ~]$ hdfs dfs -count /
          70           48             909834 /
[hadoop@node3 ~]$ hdfs dfs -count -q /
9223372036854775807 9223372036854775689            none             inf           70           48             909834 /

-mv

• Format: -mv <source path> <destination path>
• Meaning analysis: move or rename
• Command example 1, moving files:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test
Found 1 items
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 15:32 /user/hadoop/test/word_count_test.txt
[hadoop@node3 testData]$ hdfs dfs -mv /user/hadoop/test/word_count_test.txt /user/hadoop/test01
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test01
Found 1 items
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 15:32 /user/hadoop/test01/word_count_test.txt

• Command example 2, rename the file:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test01
Found 1 items
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 15:32 /user/hadoop/test01/word_count_test.txt
[hadoop@node3 testData]$ hdfs dfs -mv /user/hadoop/test01/word_count_test.txt /user/hadoop/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test01
Found 1 items
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 15:32 /user/hadoop/test01/word_count_test01.txt

-cp

• Format: -cp <source path> <destination path>
• Meaning analysis: copy
• Command example:

[hadoop@node3 testData]$ hdfs dfs -cp /user/hadoop/test01/word_count_test01.txt /user/hadoop/test/word_count_test02.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/
Found 1 items
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 15:56 /user/hadoop/test/word_count_test02.txt

-rm

• Use format: -rm [-skipTrash] path
• Meaning analysis: Delete files/blank folders (move to the trash recycle bin), add -skipTrash to completely delete from the trash recycle bin, especially useful for deleting some large files
• Command example:

[hadoop@node3 testData]$ hdfs dfs -rm /user/hadoop/test/word_count_test02.txt
Deleted /user/hadoop/test/word_count_test02.txt

-rmr

• Use format: -rmr [-skipTrash] path or -rm -r [-skipTrash] path
• Meaning analysis: delete, recursive delete
• Command example:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/t_01/t_02/t_03
[hadoop@node3 testData]$ hdfs dfs -rmr /user/hadoop/test/t_01
rmr: DEPRECATED: Please use '-rm -r' instead.
Deleted /user/hadoop/test/t_01
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/
[hadoop@node3 testData]$ 

-put

• Use format: -put [-f] <multiple files on linux> <hdfs path>
• Meaning analysis: upload files to hdfs, add -f to force upload, if the hdfs file already exists with the same name, it will be overwritten.
• Command example:

[hadoop@node3 testData]$ hdfs dfs -put word_count_test.txt echo_test.sh /user/hadoop/test/test01
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01
Found 2 items
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt

-getmerge

• Format: -getmerge <source hdfs path> <linux path>
• Meaning analysis: merge files to local
• Command example:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01
Found 5 items
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
drwxr-xr-x   - hadoop supergroup          0 2022-07-22 17:14 /user/hadoop/test/test01/test02
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt
-rwxr--r--   3 hadoop hadoop             69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:43 /user/hadoop/test/test01/word_count_test03.txt
[hadoop@node3 testData]$ hdfs dfs -getmerge /user/hadoop/test/test01 ./test01_getmerge.txt
[hadoop@node3 testData]$ ll
总用量 20
-rw-rw-r--. 1 hadoop hadoop  87 7月  21 17:00 echo_test.sh
-rw-r--r--. 1 hadoop hadoop 294 7月  22 17:40 test01_getmerge.txt
-rw-rw-r--. 1 hadoop hadoop  69 7月  22 16:11 word_count_test01.txt
-rw-rw-r--. 1 hadoop hadoop  69 7月  22 16:11 word_count_test02.txt
-rw-rw-r--. 1 hadoop hadoop  69 7月  21 14:55 word_count_test.txt
[hadoop@node3 testData]$ cat test01_getmerge.txt 
#!/bin/bash
user=`hdfs dfs -cat /user/hadoop/wordcount/input/echo_test.txt`
echo $user
hello world leo825
hello world hadoop
hello that girl
hello that boy
hello world leo825
hello world hadoop
hello that girl
hello that boy
hello world leo825
hello world hadoop
hello that girl
hello that boy

-copyFromLocal

• Use format: -copyFromLocal <multiple files on linux> <hdfs path>
• Meaning analysis: copy from local
• Command example:

[hadoop@node3 testData]$ hdfs dfs -copyFromLocal /home/hadoop/testData/word_count_test01.txt /user/hadoop/test/test01
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01
Found 3 items
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt

-moveFromLocal

• Use format: -moveFromLocal <multiple files on linux> <hdfs path>
• Meaning analysis: move local files to HDFS (equivalent to cutting)
• Command example:

[hadoop@node3 testData]$ hdfs dfs -moveFromLocal /home/hadoop/testData/word_count_test03.txt /user/hadoop/test/test01/
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/
Found 4 items
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:43 /user/hadoop/test/test01/word_count_test03.txt

-cat

• Format: -cat <hdfs file path>
• Meaning analysis: view the content of the file
• Command example:

[hadoop@node3 testData]$ hdfs dfs -cat /user/hadoop/test/test01/word_count_test.txt
hello world leo825
hello world hadoop
hello that girl
hello that boy

-text

• Format: -text <hdfs file path>
• Meaning analysis: view the content of the file
• Command example:

[hadoop@node3 testData]$ hdfs dfs -text /user/hadoop/test/test01/word_count_test.txt
hello world leo825
hello world hadoop
hello that girl
hello that boy

-tail

• Format: -tail [-f] <hdfs file path>
• Meaning analysis: view the tail information of the file
• Command example:

[hadoop@node3 testData]$ hdfs dfs -tail -f /user/hadoop/test/test01/word_count_test01.txt
hello world leo825
hello world hadoop
hello that girl
hello that boy

-copyToLocal

• Format: -copyToLocal [-ignoreCrc] [-crc] <hdfs source path> <linux destination path>
• Meaning analysis: copy the hdfs file to the local
• Command example:

[hadoop@node3 testData]$ hdfs dfs -copyToLocal /user/hadoop/test/test01/word_count_test03.txt ./
[hadoop@node3 testData]$ ll
总用量 20
-rw-rw-r--. 1 hadoop hadoop 87 7月  21 17:00 echo_test.sh
-rw-rw-r--. 1 hadoop hadoop 69 7月  22 16:11 word_count_test01.txt
-rw-rw-r--. 1 hadoop hadoop 69 7月  22 16:11 word_count_test02.txt
-rw-r--r--. 1 hadoop hadoop 69 7月  22 16:52 word_count_test03.txt
-rw-rw-r--. 1 hadoop hadoop 69 7月  21 14:55 word_count_test.txt

-mkdir

• Use format: -mkdir <hdfs path>
• Meaning Analysis: To create a blank file, only one level can be created. If you want to create a multi-level directory, you need to add the parameter -p
• Command example:

[hadoop@node3 testData]$ hdfs dfs -mkdir -p /user/hadoop/test/test01/test02/test03
[hadoop@node3 testData]$ hdfs dfs -ls -R /user/hadoop/test/test01/
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
drwxr-xr-x   - hadoop supergroup          0 2022-07-22 17:14 /user/hadoop/test/test01/test02
drwxr-xr-x   - hadoop supergroup          0 2022-07-22 17:14 /user/hadoop/test/test01/test02/test03
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:43 /user/hadoop/test/test01/word_count_test03.txt

-touchz

• Format: -touchz <hdfs file path>
• Meaning analysis: create a blank file
• Command example:

[hadoop@node3 testData]$ hdfs dfs -touchz /user/hadoop/test/test01/test02/test03/test001.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/test02/test03/test001.txt
-rw-r--r--   3 hadoop supergroup          0 2022-07-22 17:18 /user/hadoop/test/test01/test02/test03/test001.txt

-stat

• Format: -stat <hdfs file path>
• Meaning analysis: display file statistics
• Command example:

[hadoop@node3 testData]$ hdfs dfs -stat /user/hadoop/test/test01/word_count_test.txt
2022-07-22 08:07:35

-comd

• Use format: -chomd [-R] <permission mode> [path]
• Meaning analysis: modify permissions
• Command example:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/word_count_test01.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
# u 表示该档案的拥有者，g 表示与该档案的拥有者属于同一个群体(group)者，o 表示其他以外的人，a 表示所有（包含上面三者）。
# 设置拥有者 rwx 读写可执行权限
[hadoop@node3 testData]$ hdfs dfs -chmod u+rwx /user/hadoop/test/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/word_count_test01.txt
-rwxr--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt

-comm

• Use format: -chomn [-R] [owner]:[belonger group] path
• Meaning analysis: modify the file owner permissions, the user is changed to a group
• Command example:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/word_count_test01.txt
-rwxr--r--   3 hadoop supergroup         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -chown hadoop:hadoop /user/hadoop/test/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/word_count_test01.txt
-rwxr--r--   3 hadoop hadoop         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt

-chgrp

• Format: -chgrp[-R] attribute group name path
• Meaning analysis: modify file grouping
• Command example:

[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01
Found 5 items
-rw-r--r--   3 hadoop supergroup         87 2022-07-22 16:07 /user/hadoop/test/test01/echo_test.sh
drwxr-xr-x   - hadoop supergroup          0 2022-07-22 17:14 /user/hadoop/test/test01/test02
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:07 /user/hadoop/test/test01/word_count_test.txt
-rwxr--r--   3 hadoop hadoop             69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt
-rw-r--r--   3 hadoop supergroup         69 2022-07-22 16:43 /user/hadoop/test/test01/word_count_test03.txt
[hadoop@node3 testData]$ hdfs dfs -chgrp hadoop /user/hadoop/test/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -ls /user/hadoop/test/test01/word_count_test01.txt
-rwxr--r--   3 hadoop hadoop         69 2022-07-22 16:12 /user/hadoop/test/test01/word_count_test01.txt

hdfs dfsadmin

# 报告当前集群的情况
hdfs dfsadmin -report 
# 安全模式。
hdfs dfsadmin -safemode  
# leave代表关闭，安全模式关闭才是读写正常的。
hdfs dfsadmin -safemode leave  
# 进入安全模式。如正在做集群维护时，就可以手动进入安全模式，维护完再离开
hdfs dfsadmin -safemode enter 
# 安全模式下，能读不能写
>Safemode is On     

hdfs fsck

// 对hdfs 做检查。如发现 Target Replicas is 3 but 1 live replica(s) 代表有副本丢失。文件权限后的数据就代表当前文件本来需有多少副本
hdfs fsck

hdfs balancer

# 集群平衡。DN1 与 DN2 间节点平衡。封装到脚本，每天晚上定执行
hdfs balancer  
# 让 每个节点磁盘使用率，减去平均磁盘使用率 < 10%
hdfs balancer -policy datanode -threshold 10  
# 磁盘平衡。DN1中 多块磁盘的平衡。需要确保参数dfs.disk.balancer.enabled=true
hdfs diskbalancer

# 执行命令: 先生成计划  再执行
hdfs diskbalancer -plan ruozedata001
# 生产后会创建一个同名.plan.json文件，执行它
hdfs diskbalancer -execute ruozedata001.plan.json

The shell writes the string to the hdfs file

Append "hello world" string to hdfs file

echo "hello world" | hdfs dfs -appendToFile - /user/hadoop/test/test01/word_count_test01.txt

Test code:

[hadoop@node3 testData]$ hdfs dfs -cat /user/hadoop/test/test01/word_count_test01.txt
hello world leo825
hello world hadoop
hello that girl
hello that boy
[hadoop@node3 testData]$ echo "hello world" | hdfs dfs -appendToFile - /user/hadoop/test/test01/word_count_test01.txt
[hadoop@node3 testData]$ hdfs dfs -cat /user/hadoop/test/test01/word_count_test01.txt
hello world leo825
hello world hadoop
hello that girl
hello that boy
hello world

The shell reads out the content of the hdfs file

#!/bin/bash
str=`hdfs dfs -cat /user/hadoop/test/test01/word_count_test01.txt`
echo $str              

Test Results:

[hadoop@node3 testData]$ sh echo_test.sh 
hello world leo825 hello world hadoop hello that girl hello that boy hello world