Java 连接 启用kerberos的Kafka

其他 2018-10-31 07:21:20 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/TXBSW/article/details/82768955

前言：

最近换了新工作，在新环境下逐步适应中，来了近三周时间，也未能申请到一套服务器用来搭建CDH集群。一直用的是别人的集群，但是别人的集群各种权限限制，CDH集群还配置了kerberos认证，大大增加了工作量与工作难度。所以能不搞Kerberos认证最好不要弄这玩意，自讨苦吃…

CM为Kafka配置Kerberos

1.实施方案前，假设下面条件满足

  • CDH集成Kerberos和Sentry成功
  • 具备root权限

2.修改kafka配置

  • 进入Kafka服务，修改ssl.client.auth配置为none
    在这里插入图片描述

  • 启用kerberos
    在这里插入图片描述

  • 修改security.inter.broker.protocol
    在这里插入图片描述

    • 最后重启kafka服务
    • 完成以上配置Kafka集群已启用Kerberos认证

Xshell 下验证kafka 生产者，消费者

[wms_dev@bi-slave1 kafka_client]$ pwd
/usr/local/kafka_client
[wms_dev@bi-slave1 kafka_client]$ ls
client.properties  jaas.conf
  • 创建jaas.conf文件
#假设在该目录下建
[root@bi-slave1 kafka_client]# pwd
/usr/local/kafka_client

#创建文件
[root@bi-slave1 kafka_client]# vi jaas.conf
KafkaClient{
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
:wq
  • 创建client.properties文件
[root@bi-slave1 kafka_client]# vi client.properties
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
:wq
初始化kerberos账号

[root@bi-slave1 kafka_client]# kinit deng_yb
Password for [email protected]:
[root@bi-slave1 kafka_client]# klist

设置环境变量

#注意这样配置环境变量只针对当前的进程有效

[root@bi-slave1 kafka_client]# export KAFKA_OPTS="-Djava.security.auth.login.config=/usr/local/kafka_client/jaas.conf"
[root@bi-slave1 kafka_client]# echo $KAFKA_OPTS
-Djava.security.auth.login.config=/usr/local/kafka_client/jaas.conf

测试Producer

kafka-console-producer --broker-list 192.xxx.xxx.xx:9092,192.xxx.xxx.xx:9092 --topic kerbero --producer.config /usr/local/kafka_client/client.properties

#输入消息

Hello World!
Hello Kerberos!

启动producer客户端过程和发消息过程没报错即代表producer客户端配置成功
测试Consumer

kafka-console-consumer --topic kerbero --from-beginning --bootstrap-server 192.xxx.xxx.xx:9092,192.xxx.xxx.xx:9092 --consumer.config client.properties

#以下消息为针对topic kerbero

Hello World!
Hello Kerberos!

Hello World!
Hello Kerberos!

启动consumer客户端过程和接受消息过程没报错即代表客户端配置成功
上述为kafka集成kerberos服务端配置和客户端的使用

java代码连接Kafka

  • 工程架构目录
    在这里插入图片描述
    新建maven工程，pom.xml添加如下
	<dependency>
	     <groupId>org.apache.kafka</groupId><groupId>org.apache.kafka</groupId>
	     <artifactId>kafka-clients</artifactId>
	     <version>2.0.0</version> 
     </dependency>

添加krb5.conf，可直接从kdc所在服务器中拷贝

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = WONHIGH.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 WONHIGH.COM = {
  kdc = bi-master
  admin_server = bi-master
  default_realm = WONHIGH.COM
  kdc = bi-slave1
 }
 WONHIGH.CN = {
   kdc = nn209003:88
   admin_server = nn209003:749
   default_realm = WONHIGH.CN
  }

 BELLE.COM = {
  kdc = test-gtp-cdh-node01
  admin_server = test-gtp-cdh-node01
  default_realm = BELLE.COM
 }
[domain_realm]
 .bi-master = WONHIGH.COM
 bi-master = WONHIGH.COM
 nn209003 = WONHIGH.CN
 test-gtp-cdh-node01 = BELLE.COM
 test-gtp-cdh-node04 = BELLE.COM

[capaths]
  WONHIGH.COM = {
    WONHIGH.CN = .
    BELLE.COM = .
  }
  • 添加keytab文件，身份验证
    wms_dev.keytab
    这个文件是别人给我的…
  • jaas.conf 文件
KafkaClient{
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="file:D:/keytab/bk/wms_dev.keytab"
  principal="[email protected]";
};

Client{
  com.sun.security.auth.module.Krb5LoginModule
  required
  useTicketCache=false
  doNotPrompt=true
  useKeyTab=true
  keyTab="file:D:/keytab/bk/wms_dev.keytab"
  principal="[email protected]"
  debug=true;
};

kerberos 登录

/**
     * 以jass.conf文件认证kerberos
     */
    public static void getKerberosJaas(){
        System.setProperty("java.security.krb5.conf",
                Thread.currentThread().getContextClassLoader().getResource("krb5.conf").getPath());
        //加载本地jass.conf文件
        System.setProperty("java.security.auth.login.config",
                Thread.currentThread().getContextClassLoader().getResource("jaas.conf").getPath());
        //加载临时jass.conf
       /* File jaasConf = KerberosUtils.configureJAAS(Thread.currentThread().getContextClassLoader()
                .getResource("wms_dev.keytab").getPath(), "[email protected]");
        System.setProperty("java.security.auth.login.config", jaasConf.getAbsolutePath());*/
        System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
        //System.setProperty("sun.security.krb5.debug","true");

    }

上面代码是我根据下面的样例改造出来的

  • 工具类，初始化kerberos环境
package deng.yb.kafka_kerberos.utils;

import java.io.File;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;

public class MyProperties extends Properties{
    private Properties properties;
    
    private static final String JAAS_TEMPLATE =
            "KafkaClient {\n"
            + "com.sun.security.auth.module.Krb5LoginModule required\n" +
              "useKeyTab=true\n" +
              "keyTab=\"%1$s\"\n" +
              "principal=\"%2$s\";\n"
            + "};";
    
    
    public MyProperties(){
        properties = new Properties();
    }
    
    public MyProperties self(){
        return this;
    }
    
    public MyProperties put(String key , String value) {
        if (properties == null) {
            properties = new Properties();
        }
        
        properties.put(key, value);
        return self();
    }
    
    public static MyProperties initKerberos(){
     return new MyProperties()
                .put(ConsumerConfig.GROUP_ID_CONFIG, "DemoConsumer")
                .put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
                        "org.apache.kafka.common.serialization.StringDeserializer")
                .put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
                        "org.apache.kafka.common.serialization.StringDeserializer")
                .put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
                .put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000")
                .put("security.protocol", "SASL_PLAINTEXT")
                .put("sasl.kerberos.service.name", "kafka");
        
    }
    
    public static MyProperties initProducer(){
         return new MyProperties()
            .put(ProducerConfig.ACKS_CONFIG, "all")
            .put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
            .put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
            .put("security.protocol", "SASL_PLAINTEXT")
            .put("sasl.kerberos.service.name", "kafka");
    }
    
    public Properties getProperties() {
        return properties;
    }
    
    //生成jaas.conf临时文件
    public static void configureJAAS(String keyTab, String principal) {
         String content = String.format(JAAS_TEMPLATE, keyTab, principal);
        
         File jaasConf = null;
         PrintWriter writer = null;
         
        try {
             
            jaasConf  = File.createTempFile("jaas", ".conf");
            writer = new PrintWriter(jaasConf);
            
            writer.println(content);
            
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        
        } finally {
            
             if (writer != null) {
                  writer.close();
             }
             
             jaasConf.deleteOnExit();
        }
        
        System.setProperty("java.security.auth.login.config", jaasConf.getAbsolutePath());
        
    }
    
}

生产者类

package deng.yb.kafka_kerberos;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;

import deng.yb.kafka_kerberos.utils.MyProperties;

public class Producer {
    //发送topic
    public static String TOPIC_NAME = "kafak2hdfs";

    public static void main(String[] args) {

        System.setProperty("java.security.krb5.conf",
                Thread.currentThread().getContextClassLoader().getResource("krb5.conf").getPath());
        //初始化jaas.conf文件
        MyProperties.configureJAAS(Thread.currentThread().getContextClassLoader().getResource("wms_dev.keytab").getPath(), "[email protected]");
        System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");

        //System.setProperty("sun.security.krb5.debug","true");
        
        //初始化kerberos环境
        MyProperties props = MyProperties.initProducer();
        
        //kafka brokers地址
        props.put(
                ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
                "bi-slave1:9092,bi-slave2:9092,bi-slave3:9092");

        org.apache.kafka.clients.producer.Producer<String, String> producer = new KafkaProducer<String, String>(
                props.getProperties());

        for (int i = 0; i < 10; i++) {

            String key = "key-" + i;

            String message = "Message-" + i;

            ProducerRecord record = new ProducerRecord<String, String>(
                    TOPIC_NAME, key, message);

            producer.send(record);

            System.out.println(key + "----" + message);

        }

        producer.close();

    }
}

消费者

package deng.yb.kafka_kerberos;

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.ProducerConfig;

import deng.yb.kafka_kerberos.utils.MyProperties;

public class Comsumer {
    private static String TOPIC_NAME = "kafak2hdfs";

    public static void main(String[] args) {

        System.setProperty("java.security.krb5.conf", Thread.currentThread()
                .getContextClassLoader().getResource("krb5.conf").getPath());
        //初始化jaas.conf文件
        MyProperties.configureJAAS(Thread.currentThread().getContextClassLoader().getResource("wms_dev.keytab").getPath(), "[email protected]");

        System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");

        MyProperties props = MyProperties.initKerberos();

        props.put(
                ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
                "bi-slave1:9092,bi-slave2:9092,bi-slave3:9092");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(
                props.getProperties());
        consumer.subscribe(Arrays.asList(TOPIC_NAME));
        /*
         * TopicPartition partition0= new TopicPartition(TOPIC_NAME, 0);
         *
         * TopicPartition partition1= new TopicPartition(TOPIC_NAME, 1);
         *
         * TopicPartition partition2= new TopicPartition(TOPIC_NAME, 2);
         */

        // consumer.assign(Arrays.asList(partition0,partition1, partition2));

        ConsumerRecords<String, String> records = null;
        
        while (true) {
            try {
                Thread.sleep(1000);

                System.out.println();
                records = consumer.poll(Long.MAX_VALUE);

                for (ConsumerRecord<String, String> record : records) {

                    System.out.println("Receivedmessage: (" + record.key()
                            + "," + record.value() + ") at offset "
                            + record.offset());

                }

            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

        /*
         * while (true){
         *
         * try {
         *
         * Thread.sleep(10000l);
         *
         * System.out.println();
         *
         * records = consumer.poll(Long.MAX_VALUE);
         *
         * for (ConsumerRecord<String, String> record : records) {
         *
         * System.out.println("Receivedmessage: (" + record.key() + "," +
         * record.value() + ") at offset " + record.offset());
         *
         * }
         *
         * } **catch** (**InterruptedException** e){
         *
         * e.printStackTrace();
         *
         * }
         */

    }
}

猜你喜欢

转载自blog.csdn.net/TXBSW/article/details/82768955
今日推荐
周排行
Grayscale的报告显示，机构投资者接受比特币
任意角度的场景文本检测论文简单总结
努力学习的意义
蒟蒻 Wendigo 表情包
Ubuntu 14.04 Install Sublime Text 3
守卫
python3中request.urlopen()和requests.get()方法的区别
花点时间顺顺Git（下）
docker安装ngnix进行挂载
spring boot 2 统一异常处理
每日归档
更多
2024-06-13(0)
2024-06-12(0)
2024-06-11(0)
2024-06-10(0)
2024-06-09(0)
2024-06-08(0)
2024-06-07(0)
2024-06-06(0)
2024-06-05(0)
2024-06-04(10)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022