hadoop单点安装以及配置【附件】

前言

       hadoop安装版本为：hadoop-2.7.1。

下载hadoop

       Linux终端：wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

配置SSH

   1.配置ssh：ssh-keygen -t rsa -P ""；

    2.密钥启用SSH访问本地计算机：cat /home/hduser_/.ssd/id_rsa.pub >> /home/hduser_/.ssh/authorized_keys；

 修改hadoop配置

    1.解压hadoop 到 usr/local/hadoop；

    2.配置hadoop环境变量到~/.bashrc：             

#Set HADOOP_HOME
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin     

     3.使配置生效：source ~/.bashrc

配置关联HDFS

     1.配置JAVA_HOME到hadoop-env.sh：vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh;

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.7.0_80

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol.  Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

      2.在$HADOOP_HOME/etc/hadoop/core-site.xml 文件配置hadoop存储数据相关参数，

        vi /usr/local/hadoop/etc/hadoop/core-site.xml,修改如下：

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <!-- 指定目录存储让hadoop存储数据文件 -->
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
    <description>Parent directory for other temporary directories.</description>
</property>
<property>
    <!-- 指定默认文件系统 -->
    <name>fs.defaultFS </name>
    <value>hdfs://localhost:54310</value>
    <description>The name of the default file system. </description>
</property>
</configuration>

     3.创建存储文件目录：mkdir -p /app/hadoop/tmp；

     4.授予权限：chmod 750 /app/hadoop/tmp

Map Reduce配置

     1.设置hadoop环境变量：vi /etc/profile.d/hadoop.sh;

export HADOOP_HOME=/usr/local/hadoop

     2.授权：chmod +x /etc/profile.d/hadoop.sh；

     3.配置mapred，先在模板复制mapred-site.xml ，再编辑：

        复制：cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml；

        编辑：vi $HADOOP_HOME/etc/hadoop/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>mapreduce.jobtracker.address</name>
    <value>localhost:54311</value>
    <description>MapReduce job tracker runs at this host and port.
    </description>
</property>
</configuration>

配置HDFS

     1,编辑hdfs-site.xml配置：vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml；      

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

<property>
    <name>dfs.replication</name>
    <value>1</value>
    <description>Default block replication.</description>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/hduser_/hdfs</value>
</property>

</configuration>

      2.创建配置指定的目录：mkdir -p /home/hduser_/hdfs

      3.授权：chmod 750 /home/hduser_/hdfs