vmare Ubuntu under Hadoop development environment to build eclipse win10 remote connections
Build and strenuous, where the first Caicai pit, after less encountered some problems, but also to build a successful, interested, need to be cautious
To the recent Hadoop, a little here under the summary hadoop development environment, because it is not operational side, while writing blog, may not be so fine, then have the opportunity to build, re-completion,
However, the installation process will be listed here have encountered problems and solutions
vmare ubuntu installation
1, the installation vmare:
This gives Web: Refer to: vmare installation
License Note: 1F04Z-6D111-7Z029-AV0Q4-3AEH8 /
You may encounter problems:
Solution:
Turn off the firewall security software for a class
If the reference is not successful installation tutorial how to do this with the latest vmare re-install Tutorial
If the installation is unsuccessful try to consider using the latest version of VMware
VMware 2017 v14.x permanent license activation key
FF31K-AHZD1-H8ETZ-8WWEZ-WUUVA
CV7T2-6WY5Q-48EWP-ZXY7X-QGUWD
I 12.x is installed prior to the adoption of a variety of problems, tangled for two days really do not, for 14.x
2. Download Ubuntu: Ubuntu Download
3, the installation vmare ubuntu:
Given URL: Under Vmare install ubuntu
Nothing to say all, focus on that problem you may encounter:
problem:
Vmware installed in the finished Ubuntu16.10, 'SMBus Host Controller not enabled ' error message appears when you start, not get into the graphical interface.
Internet search, the solution is to enter the terminal window, edit blacklist.conf file in a graphical interface, the ban i2c_piix4 driven load.
But now the system has not finished loading, get in a terminal window, how to handle it?
Solution is as follows (pro-test available):
Virtual machine running on Linux kernel version 4.7 or higher systems during installation or at startup because of load intel_powerclamp drive and cause a crash.
Solution:
1. In the startup of Ubuntu hold the left Shift key;
The GRUB boot option 2.Ubuntu boot;
3. Then select Advanced Options of Ubuntu (Advanced Options for Ubuntu);
4. Then enter recovery of Ubuntu mode (the Ubuntu, with the Linux 4.7.xx-Generic (Recovery mODE));
5. the root into the root shell selected in the Recovery Menu;
6. enter the following command:
Code:
mount -o remount,rw /7. Open blacklist.conf
Code:
vi /etc/modprobe.d/blacklist.conf
8. In the file add the line
Code:
blacklist intel_powerclamp
9. Restart
Code:
reboot
Above can not solve:
Solution 2:
piix4_smbus ****host smbus controller not enabled
解决方法：
#root@Nagios ~# lsmod | grep i2c_piix4
i2c_piix4 12574 0
i2c_core 31274 1 i2c_piix4
#root@Nagios ~:vi /etc/modprobe.d/blacklist.conf
blacklist i2c_piix4
#root@Nagios ~:reboot
After the installation is complete: the screen is too small will find:
How to solve:
Installation Vmare Tool: installation VMareTool
Operation would not elaborate: Because I have it installed, if not installed, will be installed VmareTool tips:
Installation Sogou input method: Sogou input method installed
1, as follows
Vmare appears like the drive interface 2. After ubuntu later on: Vmtool installation
Issue to note is that it drives which are assigned to the desktop after extracting tar.gz
Then run:
vmware-install.pl
Installation.
You can solve adaptive problems are local screen size.
If there are problems:
vmare install Ubuntu can not enter the installation interface
The solution:
to manage identities open cmd window run: netsh winsock reset, reboot the system into effect, after which you can enter the installation interface
If the Winsock protocol on one machine configuration there are problems will lead to network connectivity issues, you need to use netsh winsock reset command to reset Winsock catalog so as to restore the network. This command can re-initialize the network environment to solve due to software conflicts, parameter error virus causes problems
If that does not https://jingyan.baidu.com/article/0320e2c1ef9f6c1b87507bf6.html click on the link to open
Under examination workStation version of the option of selecting what ubuntu 16.04 WorkStation 10 to 11 compatible
Do not install the virtual machine process insert U disk
ubuntu install hadoop
Okay environment: ubuntu install hadoop: Ubuntu install hadoop
Jdk address: the JDK address download and me jdk 1.8 1.6 1.8 difference, the time will be summarized
hadoop Download: hadoop Download here Download 2.7.3
Problem:
ubuntu install openssh-server-dependent errors reported settlement process
下列软件包有未满足的依赖关系：
openssh-server
: 依赖: openssh-client (= 1:6.6p1-2ubuntu1)
E:
无法修正错误，因为您要求某些软件包保持现状，就是它们破坏了软件包间的依赖关系。
sudo apt-get install openssh-client=1:6.6p1-2ubuntu1
问题处理1： Vmare ubuntu 这个IP改成虚拟机真实的
#127.0.0.1 localhost
#127.0.1.1 wangxiaoming-virtual-machine
192.168.204.128 localhost
192.168.204.128 wangxiaoming-virtual-machine
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
根据上面的连接会安装成功：但是也有问题 hadoop shell 命令：hadoop shell 命令
正确应该是：
hdfs dfs -mkdir -p /user/hadoop
-p 前面少了 -mkdir
mkdir 是创建文件夹，如果你想看你创建都文件夹，进行如下操作：hdfs dfs -ls /user/ 这个注意 最后使用绝对路径， 否则或包车，no directory
eclipse 配置hadoop 博主eclipse 版本是4.4.1
eclipse 安装hadoop 插件并且配置连接hadoop:eclipse连接hadoop
如果你不想用ant 去编译 githup 上hadoop 源码的话：
需要几个文件：
hadoop- eclipse 插件：hadoop eclipse 插件
下载之后放到eclipse 下eclipse plugins 文件夹下
把下载的Hadoop 2.7.3 在 Windows上解压后，参照博客：hadoop配置eclipse
重启eclipse ，你的eclipse 出现map reduce 表示成功。
到 下载：
具体操作，见博客：dll文件放置
除了这个配置，还要讲hadoop.dll 放到 system32 下
否则会报错：
D:\tmp\hadoop-DELL\mapred\staging\DELL6563393 null配置eclipse:eclipse 配置
简明eclipse hadoop配置：简明hadoop配置
具体我就不说了：
搭建好了可看下：
./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.5.jar wordcount input output <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hadoop="http://www.springframework.org/schema/hadoop"
xsi:schemaLocation="http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd">
<!-- 配置Configuration file-system-uri="${hd.fs}" -->
<hadoop:configuration id="configuration" resources ="file:/CPCN/Payment/StatementExternalBatch/config/core-site.xml,file:/CPCN/Payment/StatementExternalBatch/config/hdfs-site.xml,file:/CPCN/Payment/StatementExternalBatch/config/yarn-site.xml,file:/CPCN/Payment/StatementExternalBatch/config/mapred-site.xml"/>
<!-- 配置FileSystem uri="${hd.fs}"-->
<hadoop:file-system user="${hd.user}" id="fileSystem" configuration-ref="configuration" />
<!-- 原始Bean的注入方式 这个方式无法设置uri 会导致一file not exit -->
<!-- <bean id="configuration" class="org.apache.hadoop.conf.Configuration">
</bean> -->
<!-- <bean id="fileSystem" class="org.apache.hadoop.fs.FileSystem" factory-method="get>
<constructor-arg name="uri" value="${hd.fs}"/>
<constructor-arg name="conf" ref="configuration"/>
<constructor-arg name="user" value="hadoop"/>
</bean> -->
<bean id="smartDfs" class="cpcn.payment.tool.lang.hdfs.SmartDfs">
<property name="fileSystem" ref="fileSystem" />
</bean>
</beans> 去掉
file:/CPCN/Payment/StatementExternalBatch/config/yarn-site.xml,file:/CPCN/Payment/StatementExternalBatch/config/mapred-site.xml
给个demo吧：
看我的配置
其次：
core-site.xml 不用localhosy 用：
具体ip 从哪来 自己 ifconfig ，我这个就不细说了 考虑安全问题，我就不贴图了
eclispse中配置：
配置2：
上面hadoop.tmp.dir 中 和core-site.xml 配置一样
hdfs-site.xml 配置：
到此就结了，可能细化要分成好几篇才能写完，大致粗的描述下。