Development environment setup:

two files

Hadoop's win installation package, and then configure the path;

Then there is a package of several configurations, and then the configuration is placed in the hadoop installation package;

====================

Solve the NullPointerException problem when MapReduce tasks are executed locally on Windows

Original March 01, 2017 11:06:25

In order to have better work efficiency in the stage of debugging MapReduce tasks, we can set the Schema of the URI to file:///, so that the MapReduce task can scope the windows local folder. When I try to do this I get the following null pointer exception

Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)

The reason is that hadoop.dll file and winutils.exe file are missing. The solution steps are:

1. Download these two files, download address: http://download.csdn.net/download/fly_leopard/9503059

2. After decompression, copy hadoop.dll to C:\Windows\System32

3. Create the environment variable HADOOP_HOME, and then copy the winutils.exe file to the ${HADOOP_HOME}/bin directory

=============================

Using Hadoop 2.6.0, when debugging the mapreduce program under eclipse, the console does not print the log when the program is running, but displays the following information:

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN hadoop See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

It means that the log4j.properties file is not configured. Although this does not affect the normal operation of the program, it is hard to avoid seeing the log.

Solution:

Copy the log4j.properties file under the /etc/hadoop/ directory under the Hadoop2.6.0 installation directory and put it under the src directory of the MapReduce project.

===========================

maven hits the runnable jar package:

Add the first plugin to the pom file

<build>

   <plugins>
       <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-assembly-plugin</artifactId>
           <configuration>
               <descriptorRefs>
                   <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
               <archive>
                   <manifest>
                       <mainClass>com.autohome.userbehaviorenhance.mr.UserBehaviorEnhancementMR</mainClass>
                   </manifest>
               </archive>
               <encoding>UTF8</encoding>
           </configuration>
       </plugin>
       <plugin>
           <artifactId>maven-source-plugin</artifactId>
           <version>2.1</version>
           <configuration>
               <attach>true</attach>
           </configuration>
           <executions>
               <execution>
                   <phase>compile</phase>
                   <goals>
                       <goal>jar</goal>
                   </goals>
               </execution>
           </executions>
       </plugin>
       <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-compiler-plugin</artifactId>
           <version>3.3</version>
           <configuration>
               <source>1.8</source>
               <target>1.8</target>
               <encoding>UTF-8</encoding>
               
           </configuration>
       </plugin>
   </plugins>
   </build>

Then add parameters to the goal parameters above run as ---> mvn:build:

clean assembly:assembly

Then, look for the jar package with the suffix -jar-with-dependencies.jar in the packaged target to run the jar package;

In fact, you can also directly use eclipse's export to print the runnable jar package.

Be sure to pay attention to whether there are filtering settings in the configuration build path source under the buildpath configuration, otherwise some files may not be entered;

In both cases, you must pay attention;

hadoop development environment settings and runnable jar package generation

Development environment setup:

====================

Solve the NullPointerException problem when MapReduce tasks are executed locally on Windows

Guess you like