[Hadoop] Hadoop installation and configuration (pseudo-distributed)

0 Preparation

Linux Java environment configuration: https://blog.csdn.net/Tiezhu_Wang/article/details/113822949
Linux close the firewall: https://blog.csdn.net/Tiezhu_Wang/article/details/113861262
firefox installation: https:/ /blog.csdn.net/Tiezhu_Wang/article/details/113385544

1 download

Official website: https://hadoop.apache.org/releases.html

or Baidu Netdisk : Link: https://pan.baidu.com/s/1XHwHfBIu3fFSnqmtuH1p_A (Extraction code: xysm)

2 Installation

Install hadoop into the /usr/local directory:

sudo tar -zxf ~/Downloads/hadoop-3.2.1.tar.gz -C /usr/local

Switch to the directory and you can see that the decompression has been completed:

modify the file permissions (here the previous "hadoop" is the user name of the system):

cd /usr/local
sudo chown -R hadoop ./hadoop-3.2.1/

3 Check if Hadoop is available

Hadoop can be used after decompression, use the following command to view the Hadoop version:

/usr/local/hadoop-3.2.1/bin/hadoop version

4 Pseudo-distributed configuration

4.1 Set hadoop environment variables

vim ~/.bashrc

Add the following environment variables:

export HADOOP_HOME=/usr/local/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

After the change is exited, the configuration will take effect:

source ~/.bashrc

Switch to any directory and check whether the environment variable is successfully configured:

cd
hadoop version

The version information is the same as above, the configuration is successful

4.2 Modify the configuration file

Hadoop pseudo-distribution needs to modify two configuration files: core-site.xml and hdfs-site.xml
core-site.xml:

cd /usr/local/hadoop-3.2.1/etc/hadoop/
gedit ./core-site.xml

Add the following configuration, save and exit:

<configuration>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>file:/usr/local/hadoop-3.2.1/tmp</value>
		<description>A base for other temporary directories.</description>
	</property>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://localhost:9000</value>
	</property>
</configuration>

As shown in the figure:

hdfs-site.xml：

cd /usr/local/hadoop-3.2.1/etc/hadoop/
gedit ./hdfs-site.xml

Add the following configuration, save and exit:

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
	<property>
		<name>dfs.namenode.name.dir</name>
		<value>file:/usr/local/hadoop-3.2.1/tmp/dfs/name</value>
	</property>
	<property>
		<name>dfs.datanode.data.dir</name>
		<value>file:/usr/local/hadoop-3.2.1/tmp/dfs/data</value>
	</property>
</configuration>

As shown in the figure:

5 Check whether the configuration is successful

After the configuration is complete, perform the formatting of the namenode:

hdfs namenode -format

The formatting is successful when you see the following prompt:

Start the NameNode and DataNode daemons:

start-dfs.sh

Then use jps to check whether the startup is successful:

you can see that the three node processes have been started, and you can also visit localhost:9870 through a browser to view the files in HDFS:

After entering, you can view the initial HDFS directory:

if the above information appears, the configuration is successful, use the following command to close the daemon:

stop-dfs.sh