Ubuntu 16.04 LTS启动Hadoop 2.6.0遇到WARN util.NativeCodeLoader: Unable to load native-hadoop library ...

        问题:

       系统:Ubuntu 16.04 LTS

        Hadoop版本:hadoop-2.6.0-cdh5.9.3

        Hadoop可以正常启动,但是每次都会报警,警告信息如下:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

        解决方案:

        ①打开文件hadoop-env.sh:

vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

        ②在①所打开的文件添加以下语句(已有的修改HADOOP_CONF_DIR后对应路径):

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3"}

注意:HADOOP_CONF_DIR后是本地Hadoop的安装路径

        ③再添加以下语句:

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

执行②与③后保存退出:

wq

        ④执行以下命令:

source hadoop-env.sh

这时候再启动Hadoop可能还会出现原来的警告,建议重启。

        执行完上述步骤后我启动Hadoop时就不会再报错了。

        这个问题我也查了很多博客,最后发现只有这样才能解决我的问题,大家视各自情况而定,不一定都相同。可以留言交流,喷子绕道

图1 部分启动、关闭信息与进程信息

        以下为完整启动Hadoop信息:

utah@hadoop001:~/hadoop/app/hadoop-2.6.0-cdh5.9.3/sbin$ ./start-dfs.sh 
19/07/18 09:59:21 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 09:59:21 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 09:59:21 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 09:59:21 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 09:59:21 DEBUG security.Groups:  Creating new Groups object
19/07/18 09:59:21 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 09:59:21 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 09:59:21 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 09:59:21 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 09:59:21 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 09:59:21 DEBUG security.UserGroupInformation: hadoop login
19/07/18 09:59:21 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 09:59:21 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 09:59:21 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 09:59:21 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 09:59:21 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 09:59:21 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-namenode-hadoop001.out
localhost: starting datanode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-datanode-hadoop001.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-secondarynamenode-hadoop001.out
19/07/18 09:59:36 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 09:59:36 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 09:59:36 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 09:59:36 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 09:59:36 DEBUG security.Groups:  Creating new Groups object
19/07/18 09:59:36 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 09:59:36 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 09:59:36 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 09:59:36 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 09:59:36 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 09:59:36 DEBUG security.UserGroupInformation: hadoop login
19/07/18 09:59:36 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 09:59:36 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 09:59:36 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 09:59:36 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 09:59:36 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 09:59:36 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)

        以下为完整关闭Hadoop信息:

utah@hadoop001:~/hadoop/app/hadoop-2.6.0-cdh5.9.3/sbin$ ./stop-dfs.sh 
19/07/18 10:01:19 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 10:01:19 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 10:01:19 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 10:01:19 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 10:01:19 DEBUG security.Groups:  Creating new Groups object
19/07/18 10:01:19 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 10:01:19 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 10:01:19 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 10:01:19 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 10:01:19 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 10:01:19 DEBUG security.UserGroupInformation: hadoop login
19/07/18 10:01:19 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 10:01:19 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 10:01:19 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 10:01:19 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 10:01:19 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 10:01:19 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/07/18 10:01:36 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 10:01:36 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 10:01:36 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 10:01:36 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 10:01:36 DEBUG security.Groups:  Creating new Groups object
19/07/18 10:01:36 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 10:01:36 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 10:01:36 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 10:01:36 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 10:01:36 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 10:01:36 DEBUG security.UserGroupInformation: hadoop login
19/07/18 10:01:36 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 10:01:36 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 10:01:36 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 10:01:36 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 10:01:36 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 10:01:36 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
发布了143 篇原创文章 · 获赞 31 · 访问量 6万+

猜你喜欢

转载自blog.csdn.net/u25th_engineer/article/details/96423659