After the previous environment configuration, we can use the hadoop environment after continuing the following configuration
4. You want to clone 2 units (first shut down node1)
Clone node1 twice in a complete clone mode. Respectively node2, node3
4.1, for the clone machine node2
Modify hostname
As shown in the figure, modify the file
Modified ip and uui
remarks: uuid is similar to a human ID, which means a unique number. Because we are copying node1, we need to change it to unique! Note that the length and format cannot be
After modification, restart once.
4.2, for the clone machine node3
To modify the host name to node3, please refer to the command of node2 to
modify the ip to 192.168.72.13 and
modify uui to be unique, not the same as node1 and node2. The only one!
5. Ensure that the cluster computers can access each other without password
5.1, node1 example (hduser)
ssh-keygen -t rsa
ssh-copy-id node1
will ask you for the first time whether to confirm the trust relationship established between the two hosts, enter yes, and then press Enter, the
first time you will be asked to enter localhost (referring to this machine) After the user's password is
successful, we can log in to test through ssh node1, and then exit (if you exit to the root login interface, ssh-copy-id node1 is unsuccessful, or ssh-keygen -t rsa is unsuccessful) ssh node1 connection
ssh-copy-id node2 means: pass the certificate of node1 to node2. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the Enter key
. The first time you are asked to enter node2( Refers to the user hduser on another virtual machine. After the user’s password is
successful, we can log in to test through ssh node2, and then exit (if you exit to the root login interface once, ssh-copy-id node2 is unsuccessful) ssh node2 Connection
ssh-copy-id node3 means: pass the certificate of node1 to node3. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the Enter key
. The first time you are asked to enter node3( Refers to the user hduser on another virtual machine) After the user's password is
successful, we can log in to test through ssh node3, and then exit (if you exit to the root login interface, ssh-copy-id node3 is unsuccessful) ssh node3 Connection
5.2, node2 example (hduser)
ssh-keygen -t rsa
ssh-copy-id node2
will ask you for the first time whether to confirm the trust relationship established between the two hosts, enter yes, and then press Enter, the
first time you will be asked to enter localhost (referring to this machine) After the user’s password is
successful, we can log in to test through ssh node2, and then exit (if you exit to the root login interface, ssh-copy-id node2 is unsuccessful, or ssh-keygen -t rsa is unsuccessful) ssh localhost connection
ssh-copy-id node1 means: pass the certificate of node1 to node1. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the enter key
. The first time you are asked to enter node1 ( Refers to the user hduser on another virtual machine. After the user’s password is
successful, we can log in and test through ssh node1, and then exit (if you exit to the root login interface once, ssh-copy-id node1 is unsuccessful) ssh node2 Connection
ssh-copy-id node3 means: pass the certificate of node1 to node3. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the Enter key
. The first time you are asked to enter node3( Refers to the user hduser on another virtual machine) After the user's password is
successful, we can log in to test through ssh node3, and then exit (if you exit to the root login interface, ssh-copy-id node3 is unsuccessful) ssh node3 Connection
5.3, node3 example (hduser)
ssh-keygen -t rsa
ssh-copy-id node3
will ask you for the first time whether to confirm the trust relationship established between the two hosts, enter yes, and then press enter, the
first time you will be asked to enter localhost (referring to this machine) After the user’s password is
successful, we can log in to test through ssh node3, and then exit (if you exit to the root login interface, ssh-copy-id node3 is unsuccessful, or ssh-keygen-t rsa is unsuccessful) ssh node3 connection
ssh-copy-id node1 means: pass the certificate of node1 to node1. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the enter key
. The first time you are asked to enter node1 ( Refers to the user hduser on another virtual machine) After the user's password is
successful, we can log in to test through ssh node1, and then exit (if you exit to the root login interface, ssh-copy-id node1 is unsuccessful) ssh node2 Connection
ssh-copy-id node2 means: pass the certificate of node1 to node3. The
first time you are asked to confirm the trust relationship established between the two hosts, enter yes, and then press the Enter key
. The first time you are asked to enter node2 ( Refers to the user hduser on another virtual machine. After the user’s password is
successful, we can log in to test through ssh node2, and then exit (if you exit to the root login interface once, ssh-copy-id node2 is unsuccessful) ssh node2 Connection
6. Initialization and startup (hduser)
6.1. Enter the hadoop directory: cd ~/hadoop
6.2. Initialization (dangerous, just once)
bin/hdfs namenode -format
Remarks: Multiple initializations will result in inconsistent versions of the master node and data node.
If you accidentally initialize multiple times, you need to clear all the data generated by the data node.
Reinitialize
7. Start, stop, report status, role
Start sbin/start-all.sh in the directory (cd ~/hadoop) to
report the status bin/hdfs dfsadmin -report
Stop sbin/stop-all.sh to
query the role played by the current computer: jps
8. Test and run the jar program ([node1] host, hduser user)
8.1. Preparation
In the /home/hduser directory, create a folder file
and put two texts in /home/hduser/file (the content is arbitrary. It is better to have some words with spaces between the words), respectively named file1.txt file2.txt (The name is uniform but the content is not empty, mainly for our practice for a while). Such as:
8.2, test
Confirm that our hadoop is running and there is no error.
Use the terminal to enter the hadoop directory (cd ~/hadoop ):
Create a directory: bin/hadoop fs -mkdir /input2
Push files: bin/hadoop fs -put ~/file/file*.txt /input2/
List files: bin/hadoop fs -ls /input2/
Perform our introductory test:
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /input2/ /output2/wordcount1
View the results of the program execution
bin/hadoop fs -cat /output2/wordcount1/*
Remarks: Possible problems.
Question 1. mkdir: Unable to create directory "/home/hduser/hadoop/logs": insufficient permissions.
Solution: sudo chown -R hduser /home/hduser/hadoop So all nodes need to be executed.
Question 2: The firewall is The web interface (50070) of hdfs in the closed state cannot be opened.
Solution: After hadoop3, the web page console path can be changed to http://192.168.240.140:9870/ to access normally