大数据专栏 | ||
---|---|---|
上一篇 | 主目录 | 下一篇 |
【前言】
Intellij IDEA连接Hadoop HDFS实现本地调试
1 在IDEA中安装Hadoop插件
插件下载地址
CSDN
GitHub
安装插件
重启IDEA,菜单栏新增Hadoop
菜单栏Hadoop–设置,填写连接参数
连接成功
每次更改文件,可能都需要以用户登录权限,比较麻烦。这个可以配置,在hdfs-site.xml来配置
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
2 操作虚拟机hdfs集群
2.1 创建Maven项目
代码结构如下:
2.2 pom.xml添加依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.shane.hadoop</groupId>
<artifactId>bigdata</artifactId>
<version>1.0-SNAPSHOT</version>
<repositories>
<repository>
<id>apache</id>
<url>http://maven.apache.org</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.3</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<excludeTransitive>false</excludeTransitive>
<stripVersion>true</stripVersion>
<outputDirectory>./lib</outputDirectory>
</configuration>
</plugin>
</plugins>
</build>
</project>
2.3 编写测试连接代码
package com.shane.connect_hdfs;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class MyFirstHDFSDemo {
public static void main(String[] args) throws Exception {
URI uri = new URI("hdfs://192.168.65.101:9000/");
Configuration conf = new Configuration();
String user = "hadoop";
FileSystem fs = FileSystem.get(uri,conf,user);
fs.copyFromLocalFile(new Path("C:\\Users\\47463\\Desktop\\temp\\test.txt"), new Path("/"));
//fs.copyFromLocalFile(new Path(args[0]), new Path(args[1]));
boolean exist = fs.exists(new Path("/test.txt"));
if(exist) {
System.out.println("success");
}else {
System.out.println("failed");
}
fs.close();
}
}