Big data learning 1: Hadoop + MacOS environment construction

1. Prepare jdk...

2、准备Hadoop：https://hadoop.apache.org/releases.html

3. Configure SSH password-free login to facilitate the management and sharing of resources on the Hadoop cluster

      a. Turn on remote login: System Preferences-->Sharing-->Select remote login

      b. Open the terminal --> enter ssh-keygen -t rsa --> enter cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys --> enter ssh localhost to test, the configuration is successful and the following message appears:

          

4. Configure Hadoop environment variables, similar to the configuration of jdk. Open the terminal and enter: vim ~/.bash_profile. Add the following information (here I put the Hadoop file directly under the document):

           

    In this way, enter ${HADOOP_HOME} to enter the Hadoop file.

5. Configure the Hadoop environment, enter ${HADOOP_HOME}/etc/hadoop, modify the following four configuration files: hadoop-env.sh, core-site.xml, hdfs-site.xml, mapred-site.xml.

hadoop-env.sh:

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

core-site.xml:

<configuration>
    
    <!-- 指定HDFS老大（namenode）的通信地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储路径 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/Users/chenbin/Documents/hadoop-3.2.0/tmp/hdfs/data</value>
    </property>

</configuration>

hdfs-site.xml:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
　　 <property>
        <name>dfs.namenode.name.dir</name>
        <value>/Users/chenbin/Documents/hadoop-3.2.0/tmp/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/Users/chenbin/Documents/hadoop-3.2.0/tmp/hdfs/data</value>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>localhost:50070</value>
    </property>
</configuration>

mapred-site.xml:

<configuration>
    
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapred.child.java.opts</name>
        <value>-Xmx4096m</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>

</configuration>

6. Initialization: Enter the terminal and enter hadoop namenode -format

      

7. Start Hadoop.

     a、sbin/start-all.sh

     b、sbin/start-dfs.sh ，sbin/start-yarn.sh

     Enter jps to view the startup results:

     

8. Verify the startup result. Whether the browser can display the following page by entering the following URL:

http://localhost:50070

http://localhost:8088