Foreword
To tell the truth, had not previously met only place one must acquire the object for global use, or say how repeated instances of the new handle have no influence on the program, but this habit is very very bad, for example, I recently to when I want to write to the file hdfs FileSystem fs this handle is obtained only once, behind all of this fs operations with an object on the line, because there kerberos authentication, if each operation FileSystem fs need to get a handle to the verification of waste time, and secondly, it is taking up little memory!
Code
package com.bodyproperty.util;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.security.UserGroupInformation;
import org.springframework.stereotype.Component;
import java.io.IOException;
/**
* @Auther: kc
* @Date: 2018/12/19 11:22
* @Description:
*/
@Component
public class HdfsUtil {
static FileSystem fs;
static{
if (System.getProperty("os.name").toLowerCase().startsWith("win")) {
// Windows系统
System.setProperty("java.security.krb5.conf", "C:/Program Files/MIT/Kerberos/krb5.ini");
} else {
// linux系统
System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");
}
Configuration conf = new Configuration();
//这里设置namenode新
conf.set("fs.defaultFS", "hdfs://test-da-shanghai:8020");
//需要增加hadoop开启了安全的配置
conf.setBoolean("hadoop.security.authorization", true);
//配置安全认证方式为kerberos
conf.set("hadoop.security.authentication", "kerberos");
//设置namenode的principal
conf.set("dfs.namenode.kerberos.principal", "nn/[email protected]");
//设置datanode的principal值为“hdfs/[email protected]”
conf.set("dfs.datanode.kerberos.principal", "dn/[email protected]");
conf.set("dfs.support.append", "true");
conf.set("dfs.client.block.write.replace-datanode-on-failure.policy", "NEVER");
conf.set("dfs.client.block.write.replace-datanode-on-failure.enable", "true");
//通过hadoop security下中的 UserGroupInformation类来实现使用keytab文件登录
UserGroupInformation.setConfiguration(conf);
//设置登录的kerberos principal和对应的keytab文件,其中keytab文件需要kdc管理员生成给到开发人员
try {
UserGroupInformation.loginUserFromKeytab("[email protected]", "file:/usr/lihong/lihong.keytab");
} catch (Exception e) {
System.out.println("身份认证异常: " + e.getMessage());
e.printStackTrace();
}
try {
fs = FileSystem.get(conf);
} catch (IOException e) {
e.printStackTrace();
}
}
public static FileSystem getFs() {
return fs;
}
}
In fact, in the middle of all do not look at the entire code verification, mainly I use static code block will be surrounded validation logic, then fs this object will only be generated once, and each call getFs () method to obtain the objects are the same fs .