hadoop启动hdfs异常 util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Record the problems encountered in installing hadoop, so that you will encounter a record of a solution later, and it is also convenient for other people to solve the same problem. After installing hadoop, start-dfs.sh reported util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable.
Literally, the local hadoop library is not suitable for the native platform. After searching online, I found that the following three problems may be caused:
Problem 1: The native-hadoop library is 32-bit, but the system is 64-bit, which makes it unusable.
Enter the lib/native directory under the hadoop directory, and use it in the linux system
file libhadoop.so.1.0.0
得到的结果是
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=2c6a0dae993e827ec637437f921b30279487049c, not stripped
You can see that the local hadoop library is 64-bit, and my computer is also 64-bit. So it should not be a problem with this version
Problem 2: The address of the dependent library is not added in /etc/profile.
I did not add this
(1). Need to be added in /etc/profile
export HADOOP_HOME=/Users/scott/hadoop-3.2.0
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
Then re-execute the file using source /etc/profile to make the configuration effective.
(2). The same configuration needs to be added in hadoop-env.sh
Note: After I finish this step, hdfs can actually be started, but the warn of the title will still be reported. But it does not affect the use.
Problem 3: libhadoop.so.1.0.0 lacks dependent libraries or glibc
linux：
Use the ldd command to check if the libhadoop.so.1.0.0 file is missing a dynamic dependency library.
If it appears
./libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libhadoop.so.1.0.0)
linux-vdso.so.1 => (0x00007fff369ff000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3caa7ea000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3caa455000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3caac1b000)
This kind of similar is the glibc version problem, you can use ldd --version to check the version.
# ldd --version
ldd (GNU libc) 2.13
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
If you find that the version is lower than the GLIBC_2.14 that just appeared above, you need to upgrade the glibc version. This is more troublesome, and there is a certain risk, I did not encounter this problem, so I did not practice this operation. You can refer to upgrade glibc version .
mac：
There is another situation, that is, some friends may have hadoop installed on the mac, I also tried this. After eliminating problem one first, problem two basically solved the problem. If you try the third question, you will find that there is no ldd command on the mac, and the Internet says to use the otool -L file, which does not work.
libhadoop.so is written in C. Need to install binutils, and then use the readelf command in binutils.
readelf -d file|grep NEED
[External link image transfer failed (img-lOWFKq2J-1566735878129)(https://user-gold-cdn.xitu.io/2019/8/25/16cc7731fa3266b6?w=2134&h=236&f= jpeg&s=129837)]
Here -d is to view the dynamic dependency library, and grep NEED is the required library.
But here is actually just to enumerate the usage of readelf
If you want to remove this warn, you can add
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
to /etc/hadoop/log4j.properties under the hadoop folder, and this warn will not be reported if you adjust the error level to ERROR Up.