[hadoop] steps to build hadoop environment on windows

Article directory

foreword

In the field of big data development, I have to talk about the traditional classic Hadoop basic computing framework. Generally, we will deploy the hadoop cluster on the server, but as a senior brick mover, our local environment also needs a development environment for developing hadoop. So, arrange a hadoop environment on windows today.
docker deployment hadoop please move

basic environment

Download hadoop installation package

http://archive.apache.org/dist/hadoop/core/hadoop-3.3.2/
insert image description here

unzip.

Download hadoop dependencies in windows

https://github.com/cdarlint/winutils
insert image description here

After the download is complete, decompress and replace all the files in the 3.2.2 directory with the files in the hadoop bin directory:
insert image description here

And copy the hadoop.dll file to the C:\Windows\System32 directory:
insert image description here

Configure environment variables

HADOOP_HOME= D:\hadoop-3.2.2
Added in Path %HADOOP_HOME%\bin
insert image description here

cmd input hadoop version
insert image description here

Indicates that the basic environment configuration is successful.

If you need to deploy the local hadoop environment (HDFS), you need to continue to operate.

Hadoop hdfs build

Create hadfs data directory

Create a data directory under the hadoop-3.2.2 directory and
create a namenode and datanode storage directory under the data directory
data\namenode
data\datanode
insert image description here

Modify JAVA dependency

Enter the directory etc\hadoop
to modify hadoop-env.cmd
set JAVA_HOME=C:\Java\jdk1.8.0_271
insert image description here

Modify the configuration file

1. In the etc\hadoop directory, modify hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>D:\\hadoop-3.2.2\\data\\namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>D:\\hadoop-3.2.2\\data\\datanode</value>
    </property>
</configuration>

2. In the etc\hadoop directory, modify core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Initialize hdfs namenode

Enter the hadoop-3.2.2\bin directory, open the "Command Prompt" as an administrator,
enter hdfs namenode -format
and see successfully, it means that the format is successful. As shown below:
insert image description here

start hdfs

Enter the hadoop-3.2.2\sbin directory
and enter start-dfs.cmd to start hdfs.
The following namenode and datanode windows appear
insert image description here