Wu Yuxiong - natural born HADOOP performing experiments study notes: pseudo-distributed single-node installation

Purpose

Learn installation configuration of java

Learning configure their own password-free login node

Learn hdfs configuration and related commands

Learn yarn configuration

Principle

1.Hadoop install Hadoop installation for a beginner is a very difficult thing, step by installing the entire cluster hadoop particularly difficult, so a quick way to learn is to install the side while studying, installation time, first set up a single pseudo-distributed nodes, and then build a fully distributed, and finally build a highly available distributed cluster If you are interested, you can also use CDH research how to build very large scale clusters. 　　Before installing, you first need to understand a concept: hadoop has three parts, hdfs, mapreduce, yarn, which is a bunch of mapreduce java jar package, no installation, and hdfs and yarn are to be installed. A dummy node in a distributed installation, is actually the hdfs namenode, secondary namenode, datanode yarn and the nodemanager, resource manager on a single node. 　　To install in a pseudo-distributed Hadoop official online presentation, we'll introduce step by step in accordance with official website.　　
　　

First install java;

Configuring current users for their own password-free login;

Install Hadoop and configure profiles.

2.Hadoop profile Description:
　　 . 1, as soon dfs.hosts recording datanode join the cluster list of machines
　　2, mapred.hosts recording tasktracker soon as the cluster join list of machines
　　3, dfs.hosts.exclude mapred.hosts.exclude be respectively comprising remove the list of machines
　　4, master record run auxiliary namenode list of machines
　　5, slave records and tasktracker machines running datanode list
　　6, hadoop-env.sh record scripts use environment variables to run hadoop
　　7, Core-Site. xml hadoop core configuration items, such as commonly used mapreduce hdfs and i / o settings
　　8, hdfs-site.xml hadoop daemon configuration items, including namenode, datanode and other auxiliary namenode
　　9, mapred-site.xml mapreduce daemon configuration items, including jobtracker and tasktracker
　　10, hadoop-metrics.properties control how the attribute metrics posted on hadoop
　　11, log4j.properties system log files, namenode audit log, log tasktracker task child process property

3. Format HDFS file system
　　before you can use, the new HDFS installation needs to be formatted. By creating storage directory and the initial version of namenode persistent data structure, formatting process will create an empty file system. Since namenode manage all file system metadata, datanode can dynamically join and leave the cluster. So the initial formatting process does not involve datanode. By the same token, when you create a file system need not specify the size, it is the number datanode there is a cluster of decisions, can be increased according to need in a very long time after the file system format.
　　Formatting HDFS is a quick operation. Hdfs user identity to run the following command:

hdfs namenode -format

4. Start and stop the daemon
　　 Hadoop own script. And you can run commands to start and stop the daemon in the entire cluster-wide. To use these scripts (in the sbin directory), you need to tell the machine what Hadoop cluster. Official documents slaves for this purpose, the document contains a list of the machine host name or IP address, each row represents a machine information. File slaves listed datanode and Node Manager machine can run. Hadoop configuration file resides in the directory, although by modifying the settings might hadoop-env.sh in HADOOP_SLAVES file somewhere else (and given a different name). Also, you do not need to distribute the file to a working node, since only run control scripts on namenode and resource managers to use it.
　　1) Run the following command as a user with HDFS HDFS can start the daemon:

start-dfs.sh

　　Namenode machines and auxiliary namenode is running Hadoop configuration by asking the host name of the machine to decide. The following command script to find the host name of namenode.

hdfs getconf -namenodes

　　By default, the command from the Find the host name of fs.defaultFS in namenode. More specifically, some of the things done start-dfs.sh script is as follows:

1. Start a namenode on each machine, these machines obtained by the execution hdfs getconf -namenodes return value determined.

2. Start a datanode slaves file on each machine listed.

3. Start a secondary namenode on each machine, these obtained by performing hdfs get conf-secondarynamenodes return value determined.

　　2) YARN daemon is started in the same way, by running the following command on the managed resource manager machine as a yarn users:

start-yarn.sh

　　In this case, the resource manager is always start-yarn.sh script runs on the consent of the machine. Clear script complete the following things:

1. Start a resource manager on the local machine.

2. Start a Node Manager on each machine slaves file listed.

　　Likewise, it also provides stop-dfs.sh and stop-yarn.sh script used to stop daemon is started by the corresponding start script.

lab environment

1. Operating system
　　manipulator 1: Linux_Centos
　　manipulator 2: Windows_7
　　manipulator 1 default user name: root, Password: 123456
　　manipulator 2 default user name: hongya, Password: 123456

Step 1: Using the connection xshell

　　1.1 into the handler 2, click xshell, create a new session.

Name: Standalone

Host: 90.10.10.42

1.2 Click "User Authentication", enter your user name and password, and click OK.

Username: root

Password: 123456

1.3 session has been established, then select and click "Connect." We can see a successful connection.

Step 2: Add Mapping, free password

　　2.1 Add Mapping.

vi /etc/hosts

2.2 password-free login.

ssh-keygen

　　Once entered, you will be prompted to create a .ssh / id_rsa, id_rsa.pub file, which is the first key, the second is a public key. The process will be asked to enter a password in order to access process without ssh password, you can directly enter.

　　The public key is written standalone certification documents.

ssh-copy-id standalone