centos7 build hadoop3. *. * Series

　　Hadoop stepped on a lot of this recently built pit, first configure the JDK wrong path (path under the root user and ordinary users different), then there are different versions of hadoop start due to an error, find online is hadoop2. *. * Version, but I installed hadoop3. *. * version, there are some different environment configuration. I hope we do not repeat the same mistakes!

 

Download the installation package hadoop

Address: http://hadoop.apache.org/

 

Installation configuration

1. Configure the host name:

　　The file delete any original host name, add your own host name, save and exit.

vi / etc / hostname

 

2. Configure hosts file:

　　The last increase in the machine's IP address and file just with a good host name, save and exit.

vi /etc/hosts

 

3. Generate SSH keys: Execute the following command, and then has been Enter. Generating SSH keys. Generated key files will be stored in /root/.ssh/ directory. Provided that you use the root account to log in and generate a ssh key.

ssh-keygen

 

4. Free-tight configuration of the machine log: executing the following command will generate a good SSH key just sends the specified machine, the machine to achieve dense Free log.

SSH- copy- the above mentioned id root @ hostname

 

The upload and decompress downloaded Hadoop installation package, upload directory can be dragged from xshell

tar -zxvf hadoop-3.1.2.tar.gz (I write here is my package name)

 

6. Configure hadoop-env.sh file: this file is written hadoop environment variables, the main change hadoop of JAVA_HOME, HADOOP_HOME and HADOOP_CONF_DIR path, save and exit after the changes are complete.

cd hadoop-3.1.2/etc/hadoop/

we hadoop-env.sh

 

7. The modified core-site.xml file: Use vi core-site.xml open file, the following configuration parameters.

. 1 <Configuration> 
 2 <-! Specifies hdfs boss, namenode address -> 
 . 3 <Property> 
 . 4 <name> fs.defaultFS </ name> 
 . 5 <value> hdfs: // hostname: 9870 < / value> 
. 6 </ Property> 
 . 7 <-! temporary files generated during operation to specify hadoop storage directory, a default configuration if the security risk does not use the / tmp directory -> 
 . 8 <Property> 
 . 9 <name> hadoop. tmp. the dir </ name> 
 10 <value> Hadoop path / tmp </ value> 
 . 11 </ Property> 
 12 is </ Configuration>

 

8. Modify hdfs-site.xml: Use vi hdfs-site.xml open file, the following configuration parameters.

1 <! - Specifies the number of copies of data stored hdfs, including their default values . 3 ->
 2      <! - If false distribution pattern, this value is 1 ->
 . 3      <Property>
 . 4          <name> DFS .replication </ name>
 . 5          <value> . 1 </ value>
 . 6      </ Property>
 . 7      <-! hdfs the permission setting operation, to false indicates any user can manipulate files on hdfs ->
 . 8      <Property>
 . 9          <name> dfs.permissions </ name>
 10          <value> to false </ value>
 . 11      </ Property>
 12 is  
13 is      <Property>
14         <name>dfs.http.address</name>
15         <value>0.0.0.0:9870</value>
16     </property>

 

9. Review mapred-site.xml: Use VI-mapred the site.xml , the following configuration parameters. (hadoop3. * which directly modify this file, hadoop2. * When there is no initial mapred-site.xml this document, some template files, mapred-site.xml.template, need to copy, and rename mapred -site.xml)

1      <property> 
2     <!--指定 mapreduce 运行在 yarn 上--> 
3         <name>mapreduce.framework.name</name> 
4         <value>yarn</value> 
5     </property> 

 

10. The modified yarn-site.xml: Use vi yarn-site.xml open file, the following configuration parameters.

. 1 <Property> 
 2      <-! Boss resoucemanager yarn in the specified address -> 
 . 3          . <Name> yarn.resourcemanager hostname </ name> 
 . 4          <value> hostname </ value> 
 . 5      </ Property> 
 . 6      <Property > 
 . 7      <-! request data in the NodeManager -> 
 . 8          <name> yarn.nodemanager.aux-Services </ name> 
 . 9          <value> mapreduce_shuffle </ value> 
 10      </ Property>

 

11. slaves configuration file: Using vi slaves to open the file, add the following configuration parameters.

hostname   # write the hostname

 

12. The configuration Hadoop environment variables: Use vi / etc / profile to open the file, the following configuration parameters. Save and exit, using the source / etc / profile command to make the configuration take effect immediately.

1  # Configure environment variable hadoop
 2 Export HADOOP_HOME is = hadoop path
 . 3 Export the PATH the PATH = $ {}: {HADOOP_HOME is $} / bin: $ {} HADOOP_HOME is / sbin

 

13. Format namenode: namenode formatted with the following command to hadoop. successfully appeared on behalf of success.

Hadoop namenode -format

 

14.hadoop3. * Version, the error will be started at this time

 

 

Because of the lack of user-defined cause, so edit start and shutdown scripts in the directory hadoop respectively 
vim sbin / start-dfs.sh 
vim sbin / stop-dfs.sh 
add content to rely on the space above: 

1. HDFS_DATANODE_USER=root  
2. HDFS_DATANODE_SECURE_USER=hdfs  
3. HDFS_NAMENODE_USER=root
4. HDFS_SECONDARYNAMENODE_USER=root 

 

vim sbin/start-yarn.sh 
vim sbin/stop-yarn.sh

Add content to rely on the space above: 

　　YARN_RESOURCEMANAGER_USER=root
　　HADOOP_SECURE_DN_USER=yarn
　　YARN_NODEMANAGER_USER=root

15. Start hadoop: Use start-all.sh start hadoop command. After startup, by jps see the process node command. If the six node address and description successful start.