Installation environment to build:
- Install Virtual Machine VMware-workstation-full-12.5.7.20721.exe, running as an administrator, define the installation location itself, the installation process software installation housekeeper public reference number
- CentOS installation: https://jingyan.baidu.com/album/0eb457e5d7b28403f0a90577.html?picindex=3 ,, during the installation of the virtual disk file into multiple (M), selected according to the software, custom installation position, to enable the root user password, user name and user permissions to start the installation. The following error appears:
Solution: https://blog.csdn.net/qq_33468857/article/details/90346550
JDK installation: https://blog.csdn.net/dhr201499/article/details/81626466
- Jdk1.8 manually download the installation package, and extract the / usr / local /
- Configuration environment variable vim / etc / profile, add the following at the end of the file and environment variables to take effect, can be installed successfully:
Hadoop installation and start-: https://www.jianshu.com/p/e925137b2aa2 , hadoop does not start up, a start can easily find a problem
- Check ip address: ifconfig
- Modify hostname: vim / etc / hostname
- All hadoopserver attention to the need to change his name
- Turn off the firewall: systemctl stop firewall; systemctl disable firewalld
- 启动./sbin/yarn-daemon.sh start resourcemanager；./sbin/hadoop-daemon.sh start datanode；./sbin/hadoop-daemon.sh start namenode
- tail -100f logs；tail -100f logs/yarn-root-resourcemanager-bigdata.cmcc.com.out
- There are four files can be started up successfully, you can log into the web page http://192.168.42.161:8088/cluster
- Start hadoop summary of:
[hadoop@bigdata ~]$ su root [root@bigdata hadoop]# ssh localhost [root@bigdata ~]#ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa [root@bigdata ~]#cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys [root@bigdata ~]#chmod 0600 ~/.ssh/authorized_keys [root@bigdata ~]#ssh localhost [root@bigdata ~]#hdfs namenode -format [root@bigdata ~]#start-dfs.sh [root@bigdata ~]#jps [root@bigdata ~]#start-yarn.sh [root@bigdata ~]#jps
PIG installation: https://blog.csdn.net/weixin_34074740/article/details/91505214
Maven installation: https://blog.csdn.net/qq_28410283/article/details/81837151
Oozie installation:
- Modify maven repository set cd / usr / local / settings.xml file under conf maven (this operation must be, otherwise Oozie compiler error)
- Oozie to manually download and unzip /usr/local/oozie5.1.0, modify environment variables vim / etc / profile, modify hadoop version oozie.pom corresponding file, execute source / etc / proflie
- The only official source, the need to compile your own package of documents generated during compilation follows the implementation mkdistro.sh -DskipTests -Puber compile in the bin directory, in the following distro / target directory will be compiled a name Oozie-5.1 .0-distro.tar.gz package, then download the source code package before deleting or modifying name.
- Compiler error, by solving several attempts failed to solve the various methods, so subsequent selection using the compiled Oozie installation, referrer https://www.cnblogs.com/30go/p/8335523.html in 2.2ooize installation and https://blog.csdn.net/mapengbo521521/article/details/43983487 portion and mounted in three https://yq.aliyun.com/articles/369823 , should be added that:
- Add the environment variable;
- Modify conf in oozie-site.xml file, a modified version of the name, hadoop corresponding directory, no spark, it will need to be deleted
- Create a folder under libext oozie, copied to libext in hadoop package and ext-2.2.zip (oozie client plug-in), additional oozie agent in hadoop user profile hdfs-site.xml
Oozie deployment:
- mysql installation and configuration, issued oozie, and create oozie database, create successful, will generate a oozie.sql file and mysql jar package into the lib directory of oozie;
- Oozie configuration environment variable;
- Start Oozie, before starting to pay attention to start hadoop
- yarn.Tar.gz files in the directory to upload Ooize HDFS, upload only once bin / oozie-setup.sh sharelib create -fs hdfs: // localhost: 8088 -locallib oozie-sharelib-oozie version -yarn.tar.gz
Creating oozie.sql file and package the project generates war package, bin / ooziedb.sh create -sqfile oozie -sqlfile oozie.sql -run bin / oozie-setup.sh prepare-war
- After the success of the process will start BootStrap, Oozie can be viewed in a browser, the port number is: 11000
- Oozie verify successful start state, execution oozie admin -oozie http://192.168.7.88:11000/oozie -status
Oozie problems encountered
- Q: After the installation is complete entrance user login interface, shown below:
A: input [1] and press Enter to read the license agreement,
[2] input, press Enter to accept the license agreement,
Enter [q], press the Enter key to exit,
Enter [yes], press Enter to confirm,
After the restart you can enter the graphical login sector can
- Q: command directly install the JDK , unable to find a few installation package
A: the download package, to manually install
- Q: installation JDK , decompression error
A: su root enter decompress
- Q: hadoop安装报错，Error: JAVA_HOME is not set and could not be found
A: Adding JavaHome path export JAVA_HOME = / usr / java / jdk1.6.0_45
- Q: hadoop After successful installation start error
A: Turn off the firewall systemctl stop firewall, start again hadoop
- Q: hadoop can not start up all files
A: Close hadoop, and a separate start
./sbin/yarn-daemon.sh start resourcemanager；
./sbin/hadoop-daemon.sh start datanode；
./sbin/hadoop-daemon.sh start namenode
- Q: Run mvn -version correctly, but mvn being given, down without dependencies
A: Review maven warehouse provided cd / usr / local / maven settings.xml file in the conf, as follows:
- Q: oozie in oozie-site.xml file configuration issues
A: add some agents, and the corresponding version number corresponds to the port
- Q: missing part jar package
A: Manually download jar package Note: After you add the required jar package labeled as war package, otherwise it would be an error to start oozie
- Q: Oozie not start up
A: Before starting oozie hadoop need to start up, and need to start JobHistorySever process, the implementation of Mr-jobhistory- daemon.sh start historysever in the bin directory
