问题:
系统:Ubuntu 16.04 LTS
Hadoop版本:hadoop-2.6.0-cdh5.9.3
Hadoop可以正常启动,但是每次都会报警,警告信息如下:
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决方案:
①打开文件hadoop-env.sh:
vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh
②在①所打开的文件添加以下语句(已有的修改HADOOP_CONF_DIR后对应路径):
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3"}
注意:HADOOP_CONF_DIR后是本地Hadoop的安装路径;
③再添加以下语句:
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"
执行②与③后保存退出:
wq
④执行以下命令:
source hadoop-env.sh
这时候再启动Hadoop可能还会出现原来的警告,建议重启。
执行完上述步骤后我启动Hadoop时就不会再报错了。
这个问题我也查了很多博客,最后发现只有这样才能解决我的问题,大家视各自情况而定,不一定都相同。可以留言交流,喷子绕道!
以下为完整启动Hadoop信息:
utah@hadoop001:~/hadoop/app/hadoop-2.6.0-cdh5.9.3/sbin$ ./start-dfs.sh
19/07/18 09:59:21 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 09:59:21 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 09:59:21 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 09:59:21 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 09:59:21 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 09:59:21 DEBUG security.Groups: Creating new Groups object
19/07/18 09:59:21 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 09:59:21 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 09:59:21 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 09:59:21 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 09:59:21 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 09:59:21 DEBUG security.UserGroupInformation: hadoop login
19/07/18 09:59:21 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 09:59:21 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 09:59:21 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 09:59:21 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 09:59:21 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 09:59:21 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-namenode-hadoop001.out
localhost: starting datanode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-datanode-hadoop001.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/utah/hadoop/app/hadoop-2.6.0-cdh5.9.3/logs/hadoop-utah-secondarynamenode-hadoop001.out
19/07/18 09:59:36 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 09:59:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 09:59:36 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 09:59:36 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 09:59:36 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 09:59:36 DEBUG security.Groups: Creating new Groups object
19/07/18 09:59:36 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 09:59:36 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 09:59:36 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 09:59:36 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 09:59:36 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 09:59:36 DEBUG security.UserGroupInformation: hadoop login
19/07/18 09:59:36 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 09:59:36 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 09:59:36 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 09:59:36 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 09:59:36 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 09:59:36 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
以下为完整关闭Hadoop信息:
utah@hadoop001:~/hadoop/app/hadoop-2.6.0-cdh5.9.3/sbin$ ./stop-dfs.sh
19/07/18 10:01:19 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 10:01:19 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 10:01:19 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 10:01:19 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 10:01:19 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 10:01:19 DEBUG security.Groups: Creating new Groups object
19/07/18 10:01:19 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 10:01:19 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 10:01:19 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 10:01:19 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 10:01:19 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 10:01:19 DEBUG security.UserGroupInformation: hadoop login
19/07/18 10:01:19 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 10:01:19 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 10:01:19 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 10:01:19 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 10:01:19 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 10:01:19 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
19/07/18 10:01:36 DEBUG util.Shell: setsid exited with exit code 0
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
19/07/18 10:01:36 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
19/07/18 10:01:36 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
19/07/18 10:01:36 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
19/07/18 10:01:36 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
19/07/18 10:01:36 DEBUG security.Groups: Creating new Groups object
19/07/18 10:01:36 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
19/07/18 10:01:36 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
19/07/18 10:01:36 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
19/07/18 10:01:36 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
19/07/18 10:01:36 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
19/07/18 10:01:36 DEBUG security.UserGroupInformation: hadoop login
19/07/18 10:01:36 DEBUG security.UserGroupInformation: hadoop login commit
19/07/18 10:01:36 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: utah
19/07/18 10:01:36 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: utah" with name utah
19/07/18 10:01:36 DEBUG security.UserGroupInformation: User entry: "utah"
19/07/18 10:01:36 DEBUG security.UserGroupInformation: UGI loginUser:utah (auth:SIMPLE)
19/07/18 10:01:36 DEBUG security.UserGroupInformation: PrivilegedAction as:utah (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)