Springboot uses hikari connection pool for Kerberos authentication to access Impala

springboot-kerberos-hikari-impala

Springboot uses hikari connection pool and performs Kerberos authentication to access the demo project of Impala

Springboot uses hikari connection pool and performs Kerberos authentication to access Impala's demo address: https://github.com/Raray-chuan/springboot-kerberos-hikari-impala

Modified Hikari source address: https://github.com/Raray-chuan/HikariCP-4.0.3

springboot版本:2.7.5
HikariCP版本:4.0.3
impala驱动版本:2.6.3
CDH集群的Hadoop版本:3.0.0-cdh6.3.2

1. Hikari source code modification to support Kerberos authentication

If you want to know why Hikari needs to be modified to support Kerberos authentication and how to modify the Hikari source code, please go to the following address to view: https://github.com/Raray-chuan/HikariCP-4.0.3 , there is a detailed readme answer in this project

If you do not want to manually modify the source code and compile, and the HikariCP version is 4.0.3, there is a compiled HikariCP-4.0.3.jar available in the project's resources directory.

2. Prepare hive data for testing

1. Create tables and import data in hive:

create database test_xichuan_db;

CREATE TABLE test_xichuan_db.test_xichuan_table (
  start_time String,
  id STRING
)
row format delimited fields terminated by ','
stored as textfile;


测试数据：sample.txt
20211107,id1
20211207,id2
20211210,id3
20211214,id4
20211220,id5
20211222,id6
20211228,id7
20211101,id7
20211227,id4
20211228,id3
20211229,id2
20211230,id1


LOAD DATA LOCAL INPATH '/home/xichuan/sample.txt'
OVERWRITE INTO TABLE test_xichuan_db.test_xichuan_table;

2. Query data in impala

select * from test_xichuan_db.test_xichuan_table;

3.Project description

This project is a very simple springboot demo project. The project structure and each annotation will not be described in detail. Moreover, the project is authenticated through principal+keytab. How to use Kerberos, please refer to: https://raray -chuan.github.io/xichuan_note/#/docs/big-data/kerberos/CDH6.3.2 integrated Kerberos

This project is very simple. We customize the dataSource spring bean in DataSourceConfig, so that when we use mybatis to perform impala query, we will automatically obtain the dataSource bean without modifying the code.

We modified the source code of Hikari and added four Kerberos parameters to the HikariConfig class, which are:

authenticationType:安全验证的类型,如果值是"kerberos"，则进行Kerberos认证，如果为null，则不进行认证
krb5FilePath:krb5.conf文件的路径
principal:principal的名称
keytabPath:对应principal的keytab的路径

In this way, when we create HikariDataSource, we can use the HikariDataSource (HikariConfig configuration) constructor to create it and pass the Kerberos parameters into it. The DataSourceConfig code is as follows:

package com.xichuan.dev.config;

import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;
import java.sql.*;

/**
 * @Author Xichuan
 * @Date 2022/11/1 15:15
 * @Description
 */
@Configuration
public class DataSourceConfig {
    
    
    private Logger logger = LoggerFactory.getLogger(DataSourceConfig.class);

    @Value("${authentication.type}")
    private String authenticationType;

    @Value("${authentication.kerberos.krb5FilePath}")
    private String krb5FilePath;

    @Value("${authentication.kerberos.principal}")
    private String principal;

    @Value("${authentication.kerberos.keytabPath}")
    private String keytabPath;

    /**
     * inint datasource
     * @return
     */
    @Bean
    public DataSource dataSource(DataSourceProperties dataSourceProperties) throws SQLException {
    
    
        HikariConfig config = new HikariConfig();
        //kerberos config
        config.setAuthenticationType(authenticationType);
        config.setKrb5FilePath(krb5FilePath);
        config.setPrincipal(principal);
        config.setKeytabPath(keytabPath);

        //jdbc and pool config
        config.setJdbcUrl(dataSourceProperties.getUrl());
        config.setDriverClassName(dataSourceProperties.getDriverClassName());
        config.setUsername(dataSourceProperties.getUsername());
        config.setPassword(dataSourceProperties.getPassword());
        config.setPoolName(dataSourceProperties.getPoolName());
        config.setReadOnly(dataSourceProperties.isReadOnly());
        config.setAutoCommit(dataSourceProperties.isAutoCommit());
        config.setMaximumPoolSize(dataSourceProperties.getMaximumPoolSize());
        //maxLifetime 池中连接最长生命周期
        config.setMaxLifetime(dataSourceProperties.getMaxLifetime());
        //等待来自池的连接的最大毫秒数 30000
        config.setIdleTimeout(dataSourceProperties.getIdleTimeout());
        //连接将被测试活动的最大时间量
        config.setValidationTimeout(dataSourceProperties.getValidationTimeout());


        HikariDataSource dataSource = new HikariDataSource(config);
        logger.info("init new dataSource: {}", dataSource);
        return dataSource;
    }
}

4. Modify the parameters in application-prod.yml to match your own test environment

# datasource and pool
# 修改为自己的impala链接地址
datasource.xichuan.url=jdbc:impala://node01:21050/;AuthMech=1;KrbRealm=XICHUAN.COM;KrbHostFQDN=node01;KrbServiceName=impala
datasource.xichuan.driver-class-name=com.cloudera.impala.jdbc41.Driver
datasource.xichuan.username=
datasource.xichuan.password=
datasource.xichuan.pool-name=xichuan-pool
datasource.xichuan.read-only=false
datasource.xichuan.auto-commit=true
datasource.xichuan.maximum-pool-size=3
# 此处是演示值，让connection的最长生存时长为35秒
datasource.xichuan.max-lifetime=35000
datasource.xichuan.idle-timeout=10000
datasource.xichuan.validation-timeout=5000


# kerberos
# 此值为null，则不进行Kerberos认证，如果此值为kerberos，则进行kerberos认证
authentication.type=kerberos
# 修改为服务器中的krb5.conf地址，或者是本地的krb5.conf地址
authentication.kerberos.krb5FilePath=D:\\development\\license_dll\\krb5.conf
# 修改为自己的principle名称
authentication.kerberos.principal=xichuan/[email protected]
# 修改为服务器中的keytab地址，或者是本地的keytab地址
authentication.kerberos.keytabPath=D:\\development\\license_dll\\xichuan.keytab

5. Project verification demonstration

1. Start the project

We can see that after the project has just started, the Hikari connection pool has performed Kerberos authentication and created three Connections

2. Access data

Access interface:http://localhost:18080/lot_operation/all_data

We can see that everything is normal when accessing access data

3. After verifying that the Connection in HikariPool is invalid, it will automatically verify Kerberos and create a new Connection

We set it datasource.xichuan.max-lifetime=35000, so every time it passes 35秒, HikariPoolthe one in it Connectionwill be invalid and a new one will be regenerated Connection. Let's check the log:

It can be seen from the logs that HikariPoolthe ones in Connectionwill expire every 35 seconds, but HikariPoolthey will pass Kerberos authentication and create new ones Connectionto add to HikariPoolthem.