HDFS client uses
The virtual machine HDFS is the most frequently used file system in the open source Hadoop platform, which can realize massive data storage and high-throughput data reading and writing. This experiment guides users how to implement common operations on HDFS through the command line client provided by HDFS. Such as file verification, file creation, upload, download, delete, copy, move, rename, permission change, file view and other operations. Before doing this, the Hadoop environment must have been set up on the virtual machine.
###1. Obtain HDFS command help
When you use it for the first time, if you don’t know how to use HDFS commands, you can get HDFS command help, directly enter hdfs in the shell and press Enter, and the corresponding help information will be displayed, as shown in the figure below. The commands shown in are basically HDFS management commands.
[hadoop@master hadoop-2.7.1]$ hdfs
2. Use of fsck command in HDFS
2.1.classpath command use
The classpath command is used, classpath can print out the classpath information of the current hdfs.
[hadoop@master hadoop-2.7.1]$ hdfs classpath
2.2 Check files with fsck command
The sck command is used, fsck is a file system verification command, and there are many options. The following introduces and demonstrates the use of some common options. Put a file from the local file system to HDFS, and then use the fsck command to verify the file. The detailed operation is as follows.
[hadoop@master local]$ cd /opt/module/hadoop-2.7.1/
[hadoop@master hadoop-2.7.1]$ echo 'Hello World!' > test
[hadoop@master hadoop-2.7.1]$ hdfs dfs -put test /user/
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test
2.3 Verify and delete damaged files
Verify and delete damaged files, the detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test -delete
2.4 Verification will display the corresponding file
When verifying the specified directory or file, the corresponding file is displayed. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ touch test1
[hadoop@master hadoop-2.7.1]$ hdfs dfs - put test1 /user/
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test1 -files
2.5 Display file block information
When verifying a specified file or directory, display the block information of the file. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test -files -blocks
2.6 block and block location information
When verifying a specified file or directory, display the file block and the location information of the block. The detailed operation is shown in the figure below.
Record the displayed blockId, such as "blk_1073741826", which will be used in the next steps. Note that blockId is a string of numbers after blk_
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user -files -blocks -locations
2.7 Location information of the datanode where the block is located in the network topology
When verifying the specified file or directory, the block information and the location information of the datanode where the block is located in the network topology are displayed. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test - files-blocks-racks
2.8 View data storage information
View the data storage policy information of the specified file or directory. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs fsck /user/test -storagepolicies
2.9 Specify block verification
Verify the specified block. The detailed operation is as shown in the figure below.
When executing the command, please replace the blkId with the real blockId (please refer to the step "2.6 block and block location information" to obtain the blockId. The format of the blockId is blk+number , there is no underscore after the number, for example: blk_1073741826)
[hadoop@master hadoop-2.7.1]$ hdfs fsck -blockId blk_1073741826
3. Use of dfs command in HDFS
3.1 Introduction to dfs command
Use of the dfs command. The dfs command is the most commonly used command in HDFS. It has many options. The basic operations of files are realized by executing dfs and adding different options, such as reading, writing, deleting, and other operations of files, all of which can be done using the dfs command. accomplish. Common options are as follows:
3.2 Use of ls option
The ls option can view the detailed information of the specified file. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/test1
3.3 Use of the cat option
The cat option can view the file content of the specified file. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -cat /user/test | head -n 10
3.4 Use of Checksum option
The checksum option can view the checksum of the specified file. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -checksum /user/test
3.5 Use of df option
df can view the used space and unused space of the current file system. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -df
3.6 Use of du option
[hadoop@master hadoop-2.7.1]$ hdfs dfs -du /user
[hadoop@master hadoop-2.7.1]$ hdfs dfs -du -h /user
[hadoop@master hadoop-2.7.1]$ hdfs dfs -du -s /user
3.7 Use of appendToFile option
The appendToFile option can append the contents of the local specified file to the specified hdfs file. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -appendToFile test /user/test
[hadoop@master hadoop-2.7.1 ]$ hdfs dfs -cat /user/test
3.8 Use of put option
The put option can write the specified local file to the specified location of HDFS. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/
[hadoop@master hadoop-2.7.1]$ touch test2
[hadoop@master hadoop-2.7.1]$ hdfs dfs -put test2 /user/
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/
3.9 Use of get options
The get option can read and write the specified file on HDFS to the specified local directory. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ rm test2
[hadoop@master hadoop-2.7.1]$ ls
[hadoop@master hadoop-2.7.1]$ hdfs dfs -get /user/test2
[hadoop@master hadoop-2.7.1]$ ls
3.10 Use of mkdir option
The mkdir option creates a directory with the specified name at the specified location in HDFS. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -mkdir /user/huawei
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user
3.11 Use of touchz option
[hadoop@master hadoop-2.7.1]$ hdfs dfs -touchz /user/huawei/taikelab
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/huawei
3. Use of the 12mv option
The mv option has two functions, one is to rename a specified file or directory in HDFS to a new name, and the other is to move a specified file or directory in HDFS to a new location [hadoop@master
hadoop -2.7.1]$ hdfs dfs -ls /user/huawei
[hadoop@master hadoop-2.7.1]$ hdfs dfs -mv /user/test /user/huawei/
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/huawei
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/
3.13 Use of cp option
The cp option can make a copy of the specified file in HDFS. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -cp /user/huawei/test /user/taike/
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/ taike/
3.14 Use of the rm option
The rm option can delete files or directories specified in HDFS (when deleting a directory, the directory must be empty, otherwise the -r option cannot be deleted), the detailed operation is shown in the figure below [hadoop@master hadoop-2.7.1
] $ hdfs dfs -rm /user/huawei/test
[hadoop@master hadoop-2.7.1]$ hdfs dfs -rm -r /user/huawei/
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user /
3.15 Use of the chown option
The chown option can change the owner and group of the specified file or directory on HDFS (an administrator is required to execute this command) [
hadoop@master hadoop-2.7.1]$ hdfs dfs -chown hadoop:hadoop /user/taike
[hadoop @master hadoop-2.7.1]$ hdfs dfs -ls /user
3.16 Use of chmod option
The chmod option can change the permission of the specified file or directory on HDFS. The detailed operation is shown in the figure below
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user
[hadoop@master hadoop-2.7.1]$ hdfs dfs - chmod 777 /user/test1
[hadoop@master hadoop-2.7.1]$ hdfs dfs -ls /user/test1
