1. Plug-in management

1. Modify the hosts file under C:\Windows\System32\drivers\etc

2. In order to solve the problem that user permissions cannot be written

(1) Add HADOOP_USER_NAME to the environment variable of the system.

(2) Change the user name of the system and keep it consistent with the user name of hadoop.

You can use one of the above methods.

3. Add the hadoop-eclipse-plugin plugin to the plugins directory

4. Start eclipse

5. Follow the steps below to operate HDFS

Second, the encoding operation HDFS

public class HDFSDemo {
Configuration conf;//Declare configuration information
FileSystem fs ;//Define file system abstract class
private FSDataOutputStream outputStream;//Define HDFS file output stream
private Writer writer;
@Before
public void begin() throws IOException{
conf = new Configuration();//Load the configuration file under src
fs = FileSystem.get(conf);//Instantiate the HDFS object
}
@After
public void end(){
try {
fs.close();//Close the file system } catch
(IOException e) {
e.printStackTrace();
}
}
/**
* create folder
* @throws IOException
*/
@Test
public void mkdir() throws IOException{
Path path = new Path("/tmp");
fs.mkdirs(path);
}
/**
* 上传文件到HDFS
* @throws IOException
*/
@Test
public void upload() throws IOException{
Path path = new Path("/tmp/test");
outputStream = fs.create(path);//实例化输出流对象
FileUtils.copyFile(new File("f:" + File.separator + "test.txt"), outputStream);
}
/**
* 查看HDFS文件列表
* @throws IOException
* @throws FileNotFoundException
*/
@Test
public void list() throws FileNotFoundException, IOException{
Path path = new Path("/tmp");
FileStatus[] fileList = fs.listStatus(path);
for (FileStatus file : fileList) {
System.out.println(file.getPath() + "-" + file.getLen() + "-" + file.getAccessTime());
}
}
/**
* 上传序列文件
* @throws IOException
*/
@SuppressWarnings("deprecation")
@Test
public void uploadSequceFile() throws IOException{
Path path = new Path("/tmp/seq");
writer = SequenceFile.createWriter(fs, conf, path, Text.class, Text.class);
File f = new File("f:" + File.separator + "sequence");
for(File file : f.listFiles()){
writer.append(new Text(file.getName()), new Text(FileUtils.readFileToString(file))); * Download the sequence file /** }
}

* @throws IOException
*/
@Test
public void downloadSeq() throws IOException{
Path path = new Path("/tmp/seq");
Reader reader = new SequenceFile.Reader(fs, path, conf);
Text key = new Text();
Text val = new Text();
while(reader.next(key, val)){
System.out.println("文件名：" + key + "，文件内容："+val);
}
}
/**
* 删除文件
* @throws IOException
*/
@Test
public void del() throws IOException{
Path path = new Path("/tmp/test");
if (fs.exists(path)) {
fs.delete(path);
}
}
}

Hadoop - Eclipse plug-in to operate HDFS

1. Plug-in management

Second, the encoding operation HDFS

Guess you like