Hadoop pseudo-distributed installation and configuration

---

1. Modify the configuration file

1. Modify the configuration file hadoop-env.sh

Modify hadoop-env.sh under the /usr/local/java/hadoop-2.7.7/etc/hadoop directory

Excuting an order:

cd /usr/local/java/hadoop-2.7.7/etc/hadoop

Type ls to view the file

Excuting an order:

sudo vi hadoop- env.sh

Modify in hadoop-env.sh

2. Modify the configuration file core-site.xml

Execute the command in the hadoop directory:

sudo vi core-site.xml

3. Modify the configuration file hdfs-site.xml
and execute the command in the hadoop directory:

sudo vi hdfs-site.xml

2. Format the NameNode

Excuting an order:

hdfs namenode -format  或者 hadoop namenode -format

Need to enter password multiple times for verification

Start the NameNode and DataNode
to execute the command:

start-dfs.sh

Check for successful startup

3. Configure the yarn service

Modify the configuration file mapred-site.xml

Execute the command in the hadoop directory
If there is no mapred-site.xml in the directory, execute the command

sudo mv mapres-site.xml.template mapred-site.xml

Then execute the command:

sudo vi mapred-site.xml

Enter the following:

Modify the configuration file yarn-site.xml

Modify the file in the hadoop directory:

Start the yarn service

start-yarn.sh

During the startup process, you need to enter a password for verification.

Enter jps to view the startup results.

4. SSH password-free login

1. cd ~/.ssh/
2. ssh-keygen -t rsa (keep pressing enter)
3. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
4. chmod 700 ~/.ssh (modify the permission of ~/.ssh to 700)
5. chmod 644 ~/.ssh/authorized_keys
6. chmod 700 ~/.ssh/id_rsa
7. chmod 777 ~/.ssh/id_rsa.pub
8. chmod 777 ~/.ssh/known_hosts
9. ssh localhost password-free login

When restarting the distributed file system service again, there is no password confirmation!

Enter the following URL in the web interface to view the information of NameNode and Datanode, or view the files in HDFS online

http://localhost:50070