Hadoop操作HDFS的相关命令（python）

Hadoop操作HDFS的相关命令（python）

本文是基于CentOS 7系统环境，搭建Hadoop集群环境，并在主节点上进行测试

• CentOS 7
• python 3.6.8
• hadoop-2.7.1

---

一、Hadoop相关命令

(1) 查看HDFS的文件结构

hadoop fs -lsr /     

(2) 新建文件夹

hadoop fs -mkdir /test_xz/input

(3) 上传本地文件到HDFS

hadoop fs -put /home/bailang/test.txt /test_xz/input

(4) 下载HDFS中的文件至本地目录

hadoop fs -get /test_xz/input/test.txt /home/bailang/ 

(5) 列出HDFS的某目录

hadoop fs -ls /test_xz  

(6) 查看HDFS上的文件

hadoop fs -cat /test_xz/input/test.txt  

(7) 删除HDFS上的文件

hadoop fs -rm /test_xz/input/test.txt  

(8) 删除HDFS上的目录

hadoop fs -rmr /test_xz/input/

(9) 查看HDFS状态

hadoop dfsadmin -report 

(10) 进入安全模式

hadoop dfsadmin -safemode enter

(11) 离开安全模式

hadoop dfsadmin -safemode leave

二、python操作HDFS

(1) 安装相关的hdfs包

pip install hdfs

(2) 读取hdfs文件内容

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
file_path = "/test_xz/input/test.txt"
lines = []
with client.read(file_path, encoding='utf-8', delimiter='\n') as reader:
	for line in reader:
    lines.append(line.strip())
print(lines)

(3) 在hdfs上创建目录

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input"
client.makedirs(hdfs_path)

(4) 返回hdfs指定目录下的文件

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input"
print(client.list(hdfs_path, status=False))

(5) 移动或者修改文件

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input"
client.rename(source_path, dst_path)

(6) 上传文件到hdfs

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input"
local_path = "/home/bailang/test.txt"
client.upload(hdfs_path, local_path, cleanup=True)

(7) 下载hdfs上的文件到本地

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input/test.txt"
local_path = "/home/bailang"
client.download(hdfs_path, local_path, overwrite=False)

(8) 以追加模式，将数据写入hdfs文件

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input/test.txt"
client.write(hdfs_path, data, overwrite=False, append=True)

(9) 以覆盖模式，将数据写入hdfs文件

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input/test.txt"
client.write(hdfs_path, data, overwrite=True, append=False)

(10) 删除hdfs中的文件

from hdfs.client import Client
client = Client("http://172.30.11.101:50070")
hdfs_path = "/test_xz/input/test.txt"
client.delete(hdfs_path)

---