Big data learning pitfalls: HADOOP_HOME and hadoop.home.dir are unset.

Problem Descriptionjava.io.FileNotFoundException
: HADOOP_HOME and hadoop.home.dir are unset.

The local remote connection to the Hadoop cluster is abnormal. The log is as follows:

java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:448)
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:419)
at org.apache.hadoop.util.Shell.(Shell.java:496)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2972)
at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2968)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:171)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
at com.demo.zhulong.utils.FileUtils.uploadToHdfs(FileUtils.java:209)
at com.demo.zhulong.controller.ImagesController.upload(ImagesController.java:151)

The log description is very clear. HADOOP_HOME and hadoop.home.dir are not set. What are these two items mainly used for? It is the Hadoop address configured in the local environment variable. Do I need to download the Windows version of Hadoop to set it up? If you are remotely connecting to a Hadoop cluster on Linux, there is no need to download and install the Windows version of Hadoop! ! !

When connecting to the Hadoop system locally and remotely, you need to configure related Hadoop variables locally, mainly including hadoop.dll and winutils.exe.

winutils：

Since hadoop is mainly written based on Linux, winutil.exe is mainly used to simulate the directory environment under Linux. When Hadoop is running under Windows or when calling a remote Hadoop cluster, this auxiliary program is required to run. winutils is a binary file in Windows that is suitable for different versions of Hadoop systems and is built on a Windows VM. The VM is used to test Hadoop-related applications in Windows systems.

Solution:
After understanding the reason, you can download the corresponding winutils according to the version of the Hadoop cluster installed.

Download address: https://github.com/steveloughran/winutils

Note:

If the same version is not available, you can choose the nearest version to download and use. If the version used in the cluster is 2.8.5, you can download and use the version file of 2.8.3.

Set the environment variable %HADOOP_HOME% to point to the directory above the BIN directory containing WINUTILS.EXE. Right now:

1. Add system variables
   
2. Copy the bin folder in the 2.8.3 folder, and the storage address is as follows:
   
3. Restart idea and respawn, the problem is solved.

Note:

There is no need to download and install the Windows version of Hadoop, you only need to introduce winutils.exe.

Reference link
https://www.itsvse.com/thread-7093-1-1.html

https://blog.csdn.net/darkdragonking/article/details/72636917
——————————————
Copyright Statement: This article is an original article by the CSDN blogger “Play Secretly” , follow the CC 4.0 BY-SA copyright agreement, please attach the original source link and this statement when reprinting.
Original link: https://blog.csdn.net/HeyShHeyou/article/details/103441110
https://blog.csdn.net/HeyShHeyou/article/details/103441110