【Hadoop一】Hadoop伪集群环境搭建

企业开发 2018-05-12 09:18:37 阅读次数: 0
 结合网上多份文档，不断反复的修正hadoop启动和运行过程中出现的问题，终于把Hadoop2.5.2伪分布式安装起来，跑通了wordcount例子。Hadoop的安装复杂性的体现之一是，Hadoop的安装文档非常多，但是能一个文档走下来的少之又少，尤其是Hadoop不同版本的配置差异非常的大。Hadoop2.5.2于前两天发布，但是它的配置跟2.5.0，2.5.1没有分别。     系统环境: Ubuntu 12.04 LTS x86_32   一、创建用户组和用户
  •  创建用户组，为系统添加一个用户组hadoop
  
sudo addgroup hadoop
 
  • 创建用户，为系统添加一个用户hadoop 
useradd -g hadoop hadoop

 

  • 使用hadoop用户登陆
su hadoop
  • 执行如下命令，生成ssh的密钥，创建的key的路径是/home/hadoop/.ssh/id_rsa 
ssh-keygen -t rsa -P ""
       
  • 将 /home/hadoop/.ssh/id_rsa.pub中的内容追加到/home/hadoop/.ssh/authorized_keys中，保存
  • 执行ssh localhost，验证无密码即可登陆
  • 执行如下命令查看当前IPv6是否禁用，1表示禁用，0表示未禁用，默认是0
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 
  •  编辑如下文件，添加三行，禁用IPv6
sudo vim /etc/sysctl.conf
  
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
 
  •  重启机器，再次查看IPv6是否禁用
  
export JAVA_HOME=/software/devsoftware/jdk1.7.0_55
export PATH=$JAVA_HOME/bin:$PATH
 
  •  编辑/etc/profile文件，设置Hadoop相关的系统变量 
export HADOOP_HOME=/home/hadoop/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$PATH
 
  • 执行如下使上面配置的系统变量生效
source /etc/profile
 
  • 将JDK设置到Hadoop的环境脚本/home/hadoop/hadoop-2.5.2/etc/hadoop/hadoop-env.sh中，追加一行 
export JAVA_HOME=/software/devsoftware/jdk1.7.0_55
  Hadoop2.5.2有四个配置文件需要配置，它们都位于/home/hadoop/hadoop-2.5.2/etc/hadoop目录下。四个文件分别是
  • core-site.xml 
  • yarn-site.xml
  • mapred-site.xml
  • hdfs-site.xml

这写配置文件中有些需要手工创建目录，有些需要根据系统的实际情况，设置hostname，hostname不能是IP或者localhost，需要在/etc/hosts中进行设置。需要补充一点，有几个文档指出，127.0.0.1最好只跟一个hostname(即Hadoop用到的)绑定，把其余的注释掉。这个究竟是否产生影响，没有测，只是按照网上的说法，只保留一个hostname

6.1 core-site.xml配置

<configuration>
<property>

      <name>hadoop.tmp.dir</name>
      <!--目录必须手动创建出来-->
      <value>/home/hadoop/data/tmp</value>

      <description>A base for other temporary directories.</description>

  </property>

<!--file system properties-->

  <property>

      <name>fs.defaultFS</name>
      
     <!--HDFS的服务地址，只能使用域名，不能设置为IP或者localhost-->
      <value>hdfs://hostname:9000</value>

  </property>
  <property>
    <!--使用Hadoop自带的so库-->
    <name>hadoop.native.lib</name>
    <value>true</value>
    <description>Should native hadoop libraries, if present, be used.</description>
</property>

</configuration>

 

 

6.2 mapred-site.xml配置

 mapred-site.xml文件默认不存在，使用cp命令从 mapred-site.xml.template拷贝一份   
cp mapred-site.xml.template mapred-site.xml
  做如下设置，   
<configuration>
  <property>
  <name>mapreduce.framework.name</name>
  <!--yarn全是小写，不是Yarn-->
  <value>yarn</value>
  </property>
 </configuration>
    

6.3 yarn-site.xml配置

<configuration>

<!-- Site specific YARN configuration properties -->   

  <property>

    <!--yarn是小写，或许大些Y也可以-->
    <name>yarn.nodemanager.aux-services</name>
    
    <!--不是mapreduce.shuffle-->

    <value>mapreduce_shuffle</value> 

  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
 </property>     

  <property>

    <description>The address of the applications manager interface in the RM.</description>       

    <name>Yarn.resourcemanager.address</name> 
   
    <!--根据实际情况，设置hostname域名-->        
    <value>hostname:18040</value>           

  </property>

  <property>

    <description>The address of the scheduler interface.</description>

    <name>Yarn.resourcemanager.scheduler.address</name> 
 
    <!--根据实际情况，设置hostname域名-->
    <value>hostname:18030</value>   

  </property>

  <property>

    <description>The address of the RM web application.</description>

    <name>Yarn.resourcemanager.webapp.address</name> 
    <!--根据实际情况，设置hostname域名-->
    <value>hostname:18088</value>   

  </property>

  <property>

    <description>The address of the resource tracker interface.</description>

    <name>Yarn.resourcemanager.resource-tracker.address</name> 

    <!--根据实际情况，设置hostname域名-->
    <value>hostname:8025</value>   

  </property>

</configuration>

 

6.4 hdfs-site.xml 配置

<configuration>

    <property>

        <name>dfs.namenode.name.dir</name>
        <!--手工创建好-->
        <value>/home/hadoop/data/hdfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>
        <!--手工创建好-->
        <value>/home/hadoop/data/hdfs/data</value>

    </property>

    <property>

        <!--HDFS文件复本数-->
        <name>dfs.replication</name>

        <value>1</value>

    </property>

</configuration>

 

七、Hadoop初始化并启动

  • 格式化Hadoop NameNode
hadoop namenode -format
 

观察日志，如果有输出中包括Storage directory /home/hadoop/data/hdfs/name has been successfully formatted，则表示格式化成功

  • 启动Hadoop
/home/hadoop/hadoop-2.5.2/sbin/start-all.sh
 
  • 使用JDK的jps检查Hadoop状态，如果是如下结果，则表示安装成功
10682 DataNode
10463 NameNode
11229 ResourceManager
24647 Jps
11040 SecondaryNameNode
11455 NodeManager
 
  •  使用netstat -anp|grep java观察Hadoop端口号使用情况
tcp        0      0 0.0.0.0:8042            0.0.0.0:*               LISTEN      11455/java      
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      11040/java      
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      10463/java      
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      11229/java      
tcp        0      0 0.0.0.0:34456           0.0.0.0:*               LISTEN      11455/java      
tcp        0      0 0.0.0.0:13562           0.0.0.0:*               LISTEN      11455/java      
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      10682/java      
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      10682/java      
tcp        0      0 0.0.0.0:8030            0.0.0.0:*               LISTEN      11229/java      
tcp        0      0 0.0.0.0:8031            0.0.0.0:*               LISTEN      11229/java      
tcp        0      0 0.0.0.0:8032            0.0.0.0:*               LISTEN      11229/java      
tcp        0      0 0.0.0.0:8033            0.0.0.0:*               LISTEN      11229/java      
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      10682/java      
tcp        0      0 0.0.0.0:8040            0.0.0.0:*               LISTEN      11455/java
 
  • 浏览NameNode、DataNode信息，可以查看HDFS状态信息
http://hostname:50070
 
  • 浏览ResourceManagered运行状态,可以浏览MapReduce任务的执行情况
http://hostname:8088
 

八、运行Hadoop自带的WordCount实例

  • 创建本地文件用于计算这个文件中的单词数
echo "My first hadoop example. Hello Hadoop in input. " > /home/hadoop/input
 
  • 创建HDFS输入目录，用于将上面的文件写入这个目录
hadoop fs -mkdir /user/hadooper
 
  • 传文件到HDFS输入目录
hadoop fs -put /home/hadoop/input /user/hadooper
 
  • 执行Hadoop自带的WordCount例子
hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output
 
  • MapReduce的过程输出
hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output
14/11/23 19:45:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/23 19:45:05 INFO input.FileInputFormat: Total input paths to process : 1
14/11/23 19:45:05 INFO mapreduce.JobSubmitter: number of splits:1
14/11/23 19:45:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1416742510596_0001
14/11/23 19:45:06 INFO impl.YarnClientImpl: Submitted application application_1416742510596_0001
14/11/23 19:45:07 INFO mapreduce.Job: The url to track the job: http://hostname:8088/proxy/application_1416742510596_0001/
14/11/23 19:45:07 INFO mapreduce.Job: Running job: job_1416742510596_0001
14/11/23 19:45:18 INFO mapreduce.Job: Job job_1416742510596_0001 running in uber mode : false
14/11/23 19:45:18 INFO mapreduce.Job:  map 0% reduce 0%
14/11/23 19:45:26 INFO mapreduce.Job:  map 100% reduce 0%
14/11/23 19:45:36 INFO mapreduce.Job:  map 100% reduce 100%
14/11/23 19:45:37 INFO mapreduce.Job: Job job_1416742510596_0001 completed successfully
14/11/23 19:45:37 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=102
		FILE: Number of bytes written=195793
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=168
		HDFS: Number of bytes written=64
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=5994
		Total time spent by all reduces in occupied slots (ms)=6925
		Total time spent by all map tasks (ms)=5994
		Total time spent by all reduce tasks (ms)=6925
		Total vcore-seconds taken by all map tasks=5994
		Total vcore-seconds taken by all reduce tasks=6925
		Total megabyte-seconds taken by all map tasks=6137856
		Total megabyte-seconds taken by all reduce tasks=7091200
	Map-Reduce Framework
		Map input records=1
		Map output records=8
		Map output bytes=80
		Map output materialized bytes=102
		Input split bytes=119
		Combine input records=8
		Combine output records=8
		Reduce input groups=8
		Reduce shuffle bytes=102
		Reduce input records=8
		Reduce output records=8
		Spilled Records=16
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=101
		CPU time spent (ms)=2640
		Physical memory (bytes) snapshot=422895616
		Virtual memory (bytes) snapshot=2055233536
		Total committed heap usage (bytes)=308281344
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=49
	File Output Format Counters 
		Bytes Written=64

 

  • 查看MapReduce的运行结果
hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop fs -cat /user/hadooper/output/part-r-00000
Hadoop	1
Hello	1
My	1
example.	1
first	1
hadoop	1
in	1
input.	1

 

九、运行Hadoop的PI程序

hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 10 10

执行结果是3.200000000000000000

十、Hadoop常见问题

1. hadoop不正常推出后，重启后，NameNode将进入Safe Mode，不能提交任务，解决办法：

 

hadoop dfsadmin -safemode leave

 

                                           

猜你喜欢

转载自bit1129.iteye.com/blog/2159176
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
周排行
Python环境安装与基础语法（1）——计算机基础知识
IMU预积分
ADAS中的LDW、FCW、BSD、LCA、ACC、AEB、APA、DMS代表的含义
B站笔试两道题
skyeye arm 硬件虚拟机环境的搭建
Web前端静态页面示例
数组-合并排序数组 II-简单
springcloud之版本问题启动报错
面向对象-------------匿名对象(六)
输入URL到页面呈现中间发生了什么？
每日归档
更多
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022