The warning message is as follows:
The cluster environment for deploying Hadoop is
Operating System CentOS 6.7
hadoop version is hadoop-2.5.2
After the cluster is set to support gzip lzo compression, the native library of hadoop should be used when reading compressed files or compressing input files. The default location of the native library is at
$HADOOP_HOME/lib/native/Linux-amd64-64 (64-bit OS)
$HADOOP_HOME/lib/native/Linux-i386-32 (32-bit OS)
The libhadoop.so file in the folder is the native library of hadoop.
If the native library does not exist, or the native library is inconsistent with the current operating system version, the following error will be reported:
11/09/20 17:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Add debugging information settings
$ export HADOOP_ROOT_LOGGER=DEBUG,console
$ hadoop fs -text /test/data/origz/access.log.gz
Enter information:
It means that the version of glibc in the system is inconsistent with the version required by libhadoop.so
Check the libc version of the system
# ll /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.5.so
The version in the system is 2.12
Upgrade glibc in your system to 2.14
Create a new directory software and download the glibc installation package to this directory
download glibc
wget http://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.bz2
Download glibc-linuxthreads
wget http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.5.tar.bz2
解压
$tar -jxvf glibc-2.14.tar.bz2
$cd glibc-2.14
$tar -jxvf ../glibc-linuxthreads-2.5.tar.bz2
$cd ..
$export CFLAGS="-g -O2"
$./glibc-2.7/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
$make
#make install
The installation may fail, the information is as follows:
Although the prompt message requires GLIBC_2.14 to be installed, the version may still be low. I replaced it with GLIBC_2.19, and then reinstalled it and it was OK.
The installation is successful as follows:
There are three points to pay attention to in the installation and compilation process:
1. Unzip glibc-linuxthreads to the glibc directory.
2. You cannot run configure in the current directory of glibc.
3. Add the optimization switch, export CFLAGS="-g -O2", otherwise an error will occur
安装完后，可以查看ls -l /lib/libc.so.6已升级
lrwxrwxrwx 1 root root 11 Apr 24 16:49 /lib64/libc.so.6 -> libc-2.9.so
测试本地库是否升级
$ export HADOOP_ROOT_LOGGER=DEBUG,console
$ hadoop fs -text /test/data/origz/access.log.gz
12/04/25 08:54:47 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
12/04/25 08:54:47 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
12/04/25 08:54:47 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/04/25 08:54:47 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
12/04/25 08:54:47 DEBUG fs.FSInputChecker: DFSClient readChunk got seqno 0 offsetInBlock 0 lastPacketInBlock false packetLen 132100
可以看到将glibc升级后不再报错，已经成功加载本地库
After installation, you can delete the software directory, because the software is not installed in this directory. . . .
Reference: http://www.linuxidc.com/Linux/2012-04/59200.htm
http://my.oschina.net/acmfly/blog/77211/
The role of ./configure, make, make install
Reference: http://www.linuxidc.com/Linux/2011-02/32211.htm