Jar package preparation
Prepare the Hadoop Jar package, here is the package file hadoop-2.7.2.tar.gz of hadoop2.7.2 under Linux (64) environment
1. Download and unzip it to a non-Chinese directory
2. Go to the share folder, find all the jar packages in it, and copy the jar packages to the _lib folder (create it yourself, name it arbitrarily)
3. Find the jar package ending with sources.jar in all jar packages, and cut it into the _source folder (useless, can be reserved)
4. Find the jar package ending with tests.jar in all jar packages, and cut it into the _test folder (useless, can be reserved)
Eclipse preparation
Copy the corresponding compiled hadoop jar package to the non-Chinese path according to the operating system of your computer, and
download the jar package to
win7
and
win10
. The difference between win10 and win7 is only the decompressed bin directory, so win10 users need to download the win7 package and keep it Just overwrite the contents of the win10 package to the bin directory of the win7 package.
1. Unzip the downloaded file to a non-Chinese directory (for example: D:\software\hadoop-2.7.2)
2. Configure the HADOOP_HOME environment variable
3. Start Eclipse and create a Java Project
4. Create the lib folder, and then add the jar package in the _lib folder prepared before (don't forget the Build Path)
5. Create a package name: com.lzl.hdfs
6. Create the HdfsClient class
public class HdfsClient {
// upload files
public static void main(String[] args) throws IOException, InterruptedException, URISyntaxException {
// 1 get the file system
Configuration configuration = new Configuration();
// configure to run on the cluster
// configuration.set("fs.defaultFS", "hdfs access path");
// FileSystem fs = FileSystem.get(configuration);
FileSystem fs = FileSystem.get(new URI("hdfs access path"), configuration, "username");
// 2 upload file
fs.copyFromLocalFile(new Path("e:/hello.txt"), new Path("/hello2.txt"));
// 3 close the resource
fs.close();
System.out.println("over");
}
}
7. Execute the program
Hadoop user name needs to be configured when running the program
When the client operates hdfs, it has a user identity. By default, the hdfs client api will take a parameter from the jvm as its own user identity:
-DHADOOP_USER_NAME=hadoop, hadoop is the user name
8. Display the output log
In the src directory of the project, create a new file, name it "log4j.properties", and fill in the file:
log4j.rootLogger=INFO, stdout log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n log4j.appender.logfile=org.apache.log4j.FileAppender log4j.appender.logfile.File=target/spring.log log4j.appender.logfile.layout=org.apache.log4j.PatternLayout log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n