2019-09-02 23:35:22
Introduction: First, I would like to Tucao their own, actually spent 4-5 nights and noon time to do it until just before successfully resolved, I also quite admire their own!
I refer in this process other people's blog, very grateful to them, I write a blog is a good habit, and he is alert to help others!
text:
1, you need to download three kinds of essential goods: Hadoop, eclipse, Hadoop-eclipse-plus
2, Hadoop, eclipse are downloaded directly from the official website, if slow download from the official website, you can use Thunder download, so hurry up. Hadoop-eclipse-plus need to search for download from GitHub, the best download Hadoop3.x-eclipse-plus
3. After installing Hadoop, eclipse, into the film mainly on the operation, compiled Hadoop-eclipse-plus-3.xx. Because I always download the latest version, so it uses Hadoop version 3.2.0, so in this article is a translation of my own Hadoop-eclipse-plus-3.2.0 version.
4. Start the unzipped Hadoop3.x-eclipse-plus archive to the next /home/hadoop/hadoop-3.2.0, will form eclipse-hadoop3x-master directory, then all of our operations are in this directory ongoing.
5. Modify the directory / src / contrib / eclipse-plugin in the build.xml file, add the code appears place is where I modified
<!----------------------- part one---------------------------------------> <!-- Override classpath to include Eclipse SDK jars --> <path id="classpath"> <pathelement location="${build.classes}"/> <!--pathelement location="${hadoop.root}/build/classes"/--> <path refid="eclipse-sdk-jars"/> <path refid="hadoop-sdk-jars"/> <!-- new add start--> <fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset> <!-- new add end--> <!----------------------- part two---------------------------------------> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">--> <echo message="contrib: ${name}"/> <javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation= "$ {} javac.deprecation" includeantruntime = "ON" <-! new new the Add - > > < CLASSPATH the refid = "CLASSPATH" /> </ the javac > </ target > <-! ----- ---------------- part three -------------------------------- ----- -
NOTE: this section needs to be added according to their needs, and requires consistent and version of the package already installed inside Hadoop correspondence, and the> to libraries.properties in the < Copy File = "$ { } hadoop.home / Share / Hadoop / Common / lib / Java-Protobuf - $ {} protobuf.version .jar " todir =" $ {} the build.dir / lib " verbose =" to true " /> <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration2-${commons-configuration2.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang3-${commons-lang3.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/javax.servlet-api-${servlet-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-io-${commons-io.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core4-${htrace.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/woodstox-core-5.0.3.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/stax2-api-3.1.4.jar" todir="${build.dir}/lib" verbose="true"/> <jar jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar" manifest="${root}/META-INF/MANIFEST.MF"> <manifest> <attribute name="Bundle-ClassPath" value="classes/, lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, lib/hadoop-mapreduce-client-common-${hadoop.version}.jar, lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/hadoop-common-${hadoop.version}.jar, lib/hadoop-hdfs-${hadoop.version}.jar, lib/protobuf-java-${protobuf.version}.jar, lib/log4j-${log4j.version}.jar, lib/commons-cli-${commons-cli.version}.jar, lib/commons-configuration2-${commons-configuration2.version}.jar, lib/commons-httpclient-${commons-httpclient.version}.jar, lib/commons-lang3-${commons-lang3.version}.jar, lib/commons-collections-${commons-collections.version}.jar, lib/jackson-core-asl-${jackson.version}.jar, lib/jackson-mapper-asl-${jackson.version}.jar, lib/slf4j-log4j12-${slf4j-log4j12.version}.jar, lib/slf4j-api-${slf4j-api.version}.jar, lib/guava-${guava.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/netty-${netty.version}.jar, lib/javax.servlet-api-${servlet-api.version}.jar, lib/commons-io-${commons-io.version}.jar, lib/woodstox-core-5.0.3.jar, lib/stax2-api-3.1.4.jar"/>
It should be noted in several places:
1. Part III added woodstox-core-5.0.3.jar and stax2-api-3.1.4.jar to solve the problem, "Eclipse started after the installation is complete, click on the Add New Location no reaction"
2. The third part of the package under the added lib must install their own Hadoop inside the same version, where the need to check themselves one by one, will meet in the next concrete operation, a very time-consuming operation
3. forget to add the above code will depends = "init, ivy-retrieve -common" get rid of, I think, "someone is explained execute the above command time, will find the appropriate jar package, if there is no will has been stop there, but actually compiles Hadoop-eclipse-plugin does not require common related packages. "
<target name="compile" unless="skip.contrib"> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">-->
4. Also note build.xlm involved Hadoop installation directory under / share / hadoop / common / lib / time path, to observe whether Hadoop path coincides, if not modified to the same, or can not properly referenced jar package .
6, modify libraries.properties files in the directory / ivy in, on request and Hadoop installation directory share / hadoop jar in the lib which version of the package can be on the line, can not be found not modify met
Here is the code with my modifications:
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#This properties file lists the versions of the various artifacts used by hadoop and components.
#It drives ivy and the generation of a maven POM
# This is the version of hadoop we are generating
hadoop.version=3.2.0
hadoop-gpl-compression.version=0.1.0
#These are the versions of our dependencies (in alphabetical order)
apacheant.version=1.7.0
ant-task.version=2.0.10
asm.version=5.0.4
aspectj.version=1.6.5
aspectj.version=1.6.11
checkstyle.version=4.2
commons-cli.version=1.2
commons-codec.version=1.11
commons-collections.version=3.2.2
commons-configuration2.version=2.1.1
commons-daemon.version=1.0.13
commons-httpclient.version=3.0.1
commons-lang3.version=3.7
commons-logging.version=1.1.13
commons-logging-api.version=1.1.13
commons-math.version=3.1.1
commons-el.version=1.0
commons-fileupload.version=1.2
commons-io.version=2.5
commons-net.version=3.6
core.version=3.1.1
coreplugin.version=1.3.2
hsqldb.version=2.3.4
ivy.version=2.1.0
jasper.version=5.5.12
jackson.version=1.9.13
#not able to figureout the version of jsp & jsp-api version to get it resolved throught ivy
# but still declared here as we are going to have a local copy from the lib folder
jsp.version=2.1
jsp-api.version=5.5.12
jsp-api-2.1.version=6.1.14
jsp-2.1.version=6.1.14
jets3t.version=0.6.1
jetty.version=6.1.26
jetty-util.version=6.1.26
jersey-core.version=1.19
jersey-json.version=1.19
jersey-server.version=1.19
junit.version=4.11
jdeb.version=0.8
jdiff.version=1.0.9
json.version=1.0
protobuf.version=2.5.0
kfs.version=0.1
netty.version=3.10.5.Final
log4j.version=1.2.17
lucene-core.version=2.3.1
htrace.version=4.1.0-incubating
mockito-all.version=1.8.5
jsch.version=0.1.45
oro.version=2.0.8
rats-lib.version=0.5.1
servlet.version=4.0.6
servlet-api.version=3.1.0
slf4j-api.version=1.7.25
slf4j-log4j12.version=1.7.25
guava.version=11.0.2
wagon-http.version=1.0-beta-2
xmlenc.version=0.52
xerces.version=1.4.4
7. compile instructions specific reference download README.md reference to the specification in eclipse-hadoop3x-master plug-extracting file, as follows
[hadoop@hadoop01 ~]$ cd /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/src/contrib/eclipse-plugin\
[hadoop@hadoop01 eclipse-plugin]$ ant jar -Dversion=3.2.0 -Dhadoop.version=3.2.0 -Declipse.home=/home/hadoop/eclipse -Dhadoop.home=/home/hadoop/hadoop-3.2.0
[hadoop@hadoop01 eclipse-plugin]$ su root
Password:
[root@hadoop01 eclipse-plugin]# ant jar -Dversion=3.2.0 -Dhadoop.version=3.2.0 -Declipse.home=/home/hadoop/eclipse -Dhadoop.home=/home/hadoop/hadoop-3.2.0
Buildfile: /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/src/contrib/eclipse-plugin/build.xml
compile:
[echo] contrib: eclipse-plugin
jar:
[jar] Building jar: /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-3.2.0.jar
BUILD SUCCESSFUL
Total time: 5 seconds
Note points: First, first README.md read, know how to compile Hadoop-eclipse-plus plug-ins; but requires root privileges, or will be error, not compile successfully, yesterday in fact, all the parameters I should set up I am not using root privileges, so there is no compile successfully.
Under Tucao: in fact, the compilation is successful, the results of the eclipse version (I downloaded the c ++) download bug that caused the eclipse has been unable to appear in Hadoop Map / Reduce tab to download eclipse-jee-linux-gtk- x86_64.tar. gz version.
8, the hadoop-eclipse-plugin-2.xxjar into the eclipse plugins directory, start eclipse
9, open the window ===> prefernces, appear Hadoop Map / Reduce tab it shows the success of the compilation
10, but to step on the bin down, and you need to set up Hadoop Map / Reduce installation of Hadoop address inside, and then follow-up operations to ensure the correct compiler, or going back to modify the parameters, recompiling.
Other emerging issues, please refer to the following address, this is my reference in the process of compiling the article, thank them again!
1】 【文章 https://www.cnblogs.com/ljy2013/articles/4417933.html
2】 【文章 https://www.cnblogs.com/zhangchao0515/p/7099002.html
【3】 文章 https://www.cnblogs.com/PurpleDream/p/4014751.html
【文章4】https://www.cnblogs.com/sissie-coding/p/9449941.html
Full attachment build.xml Code:
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project default="jar" name="eclipse-plugin"> <import file="../build-contrib.xml"/> <path id="eclipse-sdk-jars"> <fileset dir="${eclipse.home}/plugins/"> <include name="org.eclipse.ui*.jar"/> <include name="org.eclipse.jdt*.jar"/> <include name="org.eclipse.core*.jar"/> <include name="org.eclipse.equinox*.jar"/> <include name="org.eclipse.debug*.jar"/> <include name="org.eclipse.osgi*.jar"/> <include name="org.eclipse.swt*.jar"/> <include name="org.eclipse.jface*.jar"/> <include name="org.eclipse.team.cvs.ssh2*.jar"/> <include name="com.jcraft.jsch*.jar"/> </fileset> </path> <path id="hadoop-sdk-jars"> <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> <include name="hadoop*.jar"/> </fileset> <fileset dir="${hadoop.home}/share/hadoop/hdfs"> <include name="hadoop*.jar"/> </fileset> <fileset dir="${hadoop.home}/share/hadoop/common"> <include name="hadoop*.jar"/> </fileset> </path> <!-- Override classpath to include Eclipse SDK jars --> <path id="classpath"> <pathelement location="${build.classes}"/> <!--pathelement location="${hadoop.root}/build/classes"/--> <path refid="eclipse-sdk-jars"/> <path refid="hadoop-sdk-jars"/> <!-- new add --> <fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset> </path> <!-- Skip building if eclipse.home is unset. --> <target name="check-contrib" unless="eclipse.home"> <property name="skip.contrib" value="yes"/> <echo message="eclipse.home unset: skipping eclipse plugin"/> </target> <target name="compile" unless="skip.contrib"> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">--> <echo message="contrib: ${name}"/> <javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation="${javac.deprecation}" includeantruntime="on"> <classpath refid="classpath"/> </javac> </target> <!-- Override jar target to specify manifest --> <target name="jar" depends="compile" unless="skip.contrib"> <mkdir dir="${build.dir}/lib"/> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/common"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/hdfs"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/yarn"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/classes" verbose="true"> <fileset dir="${root}/src/java"> <include name="*.xml"/> </fileset> </copy> <copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration2-${commons-configuration2.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang3-${commons-lang3.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/javax.servlet-api-${servlet-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-io-${commons-io.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core4-${htrace.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/woodstox-core-5.0.3.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/stax2-api-3.1.4.jar" todir="${build.dir}/lib" verbose="true"/> <jar jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar" manifest="${root}/META-INF/MANIFEST.MF"> <manifest> <attribute name="Bundle-ClassPath" value="classes/, lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, lib/hadoop-mapreduce-client-common-${hadoop.version}.jar, lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/hadoop-common-${hadoop.version}.jar, lib/hadoop-hdfs-${hadoop.version}.jar, lib/protobuf-java-${protobuf.version}.jar, lib/log4j-${log4j.version}.jar, lib/commons-cli-${commons-cli.version}.jar, lib/commons-configuration2-${commons-configuration2.version}.jar, lib/commons-httpclient-${commons-httpclient.version}.jar, lib/commons-lang3-${commons-lang3.version}.jar, lib/commons-collections-${commons-collections.version}.jar, lib/jackson-core-asl-${jackson.version}.jar, lib/jackson-mapper-asl-${jackson.version}.jar, lib/slf4j-log4j12-${slf4j-log4j12.version}.jar, lib/slf4j-api-${slf4j-api.version}.jar, lib/guava-${guava.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/netty-${netty.version}.jar, lib/javax.servlet-api-${servlet-api.version}.jar, lib/commons-io-${commons-io.version}.jar, lib/woodstox-core-5.0.3.jar, lib/stax2-api-3.1.4.jar"/> </manifest> <fileset dir="${build.dir}" includes="classes/ lib/"/> <!--fileset dir="${build.dir}" includes="*.xml"/--> <fileset dir="${root}" includes="resources/ plugin.xml"/> </jar> </target> </project>
附README.md
hadoop3x-eclipse-plugin ======================= eclipse plugin for hadoop 3.x.x How to build ---------------------------------------- [hdpusr@demo hadoop2x-eclipse-plugin]$ cd src/contrib/eclipse-plugin # Assume hadoop installation directory is /usr/share/hadoop [hdpusr@apclt eclipse-plugin]$ ant jar -Dversion=3.1.0 -Dhadoop.version=3.1.0 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/share/hadoop final jar will be generated at directory ${hadoop2x-eclipse-plugin}/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.4.1.jar release version included ------------------------------------- release/hadoop-eclipse-kepler-plugin-2.4.1.jar # not tested yet release/hadoop-eclipse-kepler-plugin-2.2.0.jar options required -------------------------------------- version: plugin version hadoop.version: hadoop version you want to compiled with eclipse.home: path of eclipse home hadoop.home: path of hadoop 3.x home How to debug -------------------------------------- start eclipse with debug parameter: /opt/eclipse/eclipse -clean -consolelog -debug Note: compile issues resolve: -------------------------------------- 1. For different hadoop, adjust ${hadoop2x-eclipse-plugin-master}/ivy/libraries.properties, to match hadoop dependency lib version. 2. modify ${hadoop3x-eclipse}/src/contrib/eclipse-plugin/build.xml, in the node: <attribute name="Bundle-ClassPath" .... to add the jar needed. 3. For eclipse, do **not** use eclipse-inst, install eclipse **FULL VERSION(SDK)** instead.