Ubuntu 20.0.4 Hadoop3.3.2 Installation and Configuration Full Process Nanny Tutorial

Ubuntu 20.0.4 Hadoop3.3.2 installation and configuration full-process nanny-level tutorial

Preparation

The compressed package I downloaded is based on the x86 instruction set of the Windows system. If you are an Apple computer, you may need to choose the ARM version
3.21 update: Some functions of hadoop do not support JDK11 and above (for example, the webpage management cannot view the file system), so choose JDK8 when downloading, and you need to log in to the Oracle account to download from the official website

1. Brand new ubuntu virtual machinehttps://ubuntu.com/download/desktop
2. JDK Compressed Package (for linux x64 Compressed Archive)https://www.oracle.com/java/technologies/downloads/
3. Hadoop compressed package (Binary download)https://hadoop.apache.org/releases.html
4. Make full use of the snapshot function of VMware virtual machines and roll back when something goes wrong
5. Your hands and the beautiful character of concentration and seriousness

1. Configure your new system

At this time, I am using a new virtual machine that has just been installed. I will use this virtual machine to re-enter all the subsequent processes and screenshots. If you already have a well-configured virtual machine, you can jump to the chapter you need in the directory.

1. Replace the domestic source and update the software

After entering the new system, you will be prompted to update the software. At this time, we choose to skip it because the speed is very slow. In order to increase the speed of software updates, it is recommended to use domestic sources, which can greatly increase the update speed. We click on all software in the lower left corner of the desktop and click on the purple icon to
enter the software and update settings. Click on the Other option
in the Download from selection box and click on Select Best Server to automatically find The best server, according to my own test, just choose here directly, no need to modify the file. After waiting for the query to complete, click Choose Server and click Reload to update dependencies . Close the setting interface and return to the desktop. Click the lower left corner again to open this time. The silver software updater updates the software, wait for the update to complete, I can run at a speed of more than ten megabits per second, and enter the installation step without taking a screenshot. After the update is completed, choose to restart immediately and the software update is complete.

2. Copy the prepared compressed package to the virtual machine

Choose a location to store the compressed package we prepared according to your own preferences. I choose to put it in the Downloads folder under the user's home folder

3. Other considerations

1. When you customize your virtual machine settings, do not change your desktop picture. If you directly find a picture from the host and put it into the virtual machine and set it as the wallpaper, then restart and log in to the system may cause a black screen. The specific reason Unknown, I reinstalled the system twice for this reason (true big pit)
2. During the installation process, if you encounter some problems such as insufficient permissions, try to use the administrator identity to execute, and do not change the ownership of the system folder at will.
3. This article will use the editor that comes with the system to modify the file. If you are not used to it, you can install a VS Code to modify the file. It is very convenient. If you have insufficient permissions, you can enter the administrator password to save the changes directly.
4. It is best to save a snapshot of the virtual machine every time a part of the configuration is completed, so that it can be rolled back in case of subsequent configuration errors

2. Install and configure the Java environment

1. Unzip the JDK compressed package and move the location

Right-click the folder where the compressed package was stored before to open the menu, and select Open in Terminal
3.21 update: Subsequent tests found that there was an error when the webpage managed hdfs files, so the JDK version was changed to 1.8.0_x

Use the command tar -zxvf jdk-17_linux-x64_bin.tar.gzto decompress the JDK compressed package (the file name behind is the compressed file name you downloaded yourself). Note that different compression formats have different decompression instructions. I will take the tar.gz compressed package
under normal circumstances as an example . After decompression, the original directory will appear. After decompression Continue to enter the command in the terminal to move the decompressed folder to /usr/java (need to enter the administrator password) to check whether the move is successful


sudo mv jdk-17.0.2 /usr/java

2. Configure environment variables

Open the Home folder, select Show Hidden Files in the upper right corner option ,

find the file named .profile , open it with a text editor, and add the following code at the end

export JAVA_HOME=/usr/java

# JDK8 添加 JRE_HOME 和 CLASSPATH 配置
export JRE_HOME=$JAVA_HOME/jre
CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:$PATH

Be careful not to have too many spaces or too few spaces. It is recommended to copy directly. After the modification is complete, click Save to save
The following picture is inconsistent with the above code, please refer to the above code

Terminal input source ~/.profileApply the environment variables we configured

Create a system soft link
sudo update-alternatives --install /usr/bin/javac javac /usr/java/bin/javac 1
sudo update-alternatives --install /usr/bin/javac java /usr/java/bin/java 1

3. Check the installation

Enter java --versionor java -versionto check whether the configuration is successful, and the following java version prompt appears to indicate that the configuration is successful
3.21 update: Java version changed to Java 1.8.0_x for compatibility with Hadoop

3. Install and configure Hadoop

1. Unzip the Hadoop archive and move the location

Let's go back to the folder where the compressed package is stored, and right-click to open in the terminal .

Use the command tar -zxvf hadoop-3.3.2.tar.gzto decompress the compressed package, and the decompressed folder will appear in the directory.
Continue to use the command sudo mv hadoop-3.3.2 /usr/hadoopto move the folder to the /usr/hadoop directory (administrator password is required )

2. Configure JDK path

Open the file manager, find /etc/hadoop/hadoop-env.sh in the Hadoop installation directory and open it with a text editor (double-click by default)
Note here is hadoop-env.sh instead of hadoop-env.cmd
Add the following code at the end of the file (it is recommended to copy directly) and save and exit

export JAVA_HOME=/usr/java

3. Configure Hadoop

Find the core-site.xml file in the directory just now , open it with a text editor, add the following configuration in the
<configuration> tag , save and exit

  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/usr/hadoop/tmp</value>
  </property>

  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>

Continue to find the hdfs-site.xml file in the folder , open it with a text editor, add the following configuration in the
<configuration> tag , save and exit

  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/usr/hadoop/tmp/dfs/name</value>
  </property>

  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/hadoop/tmp/dfs/data</value>
  </property>

  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:50070</value>
  </property>

4. start

Open any terminal, enter the /hadoop/bin path cd /usr/hadoop/bin
, execute the command, ./hdfs namenode -formatformat it

, enter the /hadoop/sbin path, cd /usr/hadoop/sbin
execute the command, ./start-all.shstart hadoop

, execute the command jps, check the running process

, open the browser input http://localhost:50070, enter the web management page

, and the Hadoop configuration is completed

4. Appendix

1. ssh password-free login settings

Install the openssh service sudo apt-get install openssh-server
Use the administrator password to verify, enter y

to log in to the ssh localhost
host when asked whether to continue Enter yes when asked to continue and then enter the administrator password to log in

Exit the host exit

Open the file manager, find the .ssh folder in Home , and enter this folder It is hidden by default, but we have enabled the display of hidden files, right-clicked after entering the .ssh folder, and selected to open in the terminal. Enter commands in the terminal to create the public key and private key . During the process, you need to press Enter several times until the creation is complete. The public and private key files continue to be input in the terminal to perform key-free login authorization. At this point, the settings are complete, and the password is no longer required to log in to the host. Copy the private key file and register it to another computer to achieve remote access


ssh-keygen -t rsa




cat ./id_rsa.pub >> ./authorized_keys

ssh localhost

2. VS Code installation and use

Do not download VSCode directly in the app store, because there will be problems that you cannot input Chinese
Official download address: https://code.visualstudio.com/Download
Download the .deb installation package and run the installation directly

Install the Chinese extension. Search Chinese
in the extension store to find the first extension and click Install to install it. After completion, click Change Language and Restart in the pop-up window in the lower right corner . After restarting, the Chinese interface can be applied. Install the Java extension. Search for java in the extension store and select Extension Pack for Java After installation, you can change the IDE in seconds and start creating a java project . The created example can be run directly, press F5