kafka authentication mode generally have the following three:
1. SASL / GSSAPI from version 0.9.0.0 began to support
2. SASL / PLAIN began to support from version 0.10.0.0
3. SASL / SCRAM-SHA-256 and SASL / SCRAM-SHA-512 began to support from version 0.10.2.0
Wherein a first SASL / GSSAPI kerberos authentication is authentication for java native support it has, but for a little trouble python is disposed below said about the specific configuration, confluent kafka module depends on the underlying librdkafka, which is c prepared using high-performance kafka client library, there are a lot of language libraries are dependent on this, so open GSSAPI interface also needs support in librdkafka compile time
librdkafka Source: https: //github.com/edenhill/librdkafka
You need to install the necessary development packages before compiling, otherwise the relevant interface compiler does not go
The first is the openssl library, use yum to install: yum -y install openssl openssl-devel , compile openssl only supports the default PLAIN there SCRAM these two mechanisms, can not support GSSAPI mechanism also needs to be compiled libsasl2 dependent, yum install command as follows:
yum install cyrus-sasl-gssapi cyrus-sasl-devel
Command in the Ubuntu: APT-GET the install libsasl2-modules GSSAPI-MIT-dev-libsasl2 installed SDK libsasl2
Then confirm whether there zlib library, this is convenient for kafka message compression used, usually exist, the install command: yum install zlib-devel , if you need higher performance can manually compile and install zstd enable compression, not detailed here narrative
The above libraries are installed successfully you can start to compile librdkafka source, the source package is here: librdkafka-1.2.1.tar.gz, install command as follows:
# Extract package the tar -xvzf librdkafka- 1.2 . . 1 . The tar .gz CD librdkafka - 1.2 . . 1 # compile the code . / Configure the make the make the install
Note after executing the above command ./configure, according to the output libssl and confirm whether libsasl2 is turned on, as follows:
Here libssl and libsasl2 show ok Description is now SSL and SASL SCRAM and SASL GSSAPI have been supported, running the configure stage did not specify the prefix of the default installation location is / usr / local, dynamic library location on is: / usr / local / lib, you need to add this directory to the dynamic library connection list, such as added /etc/ld.so.conf, execute ldconfig take effect after saving
Finally, you can compile and install the confluent kafka python module, where the installed version is 1.2.0, you can run the following code testing after installation:
1 #!/usr/bin/env python3 2 # coding=utf-8 3 from confluent_kafka import Producer 4 5 def delivery_report(err, msg): 6 """ Called once for each message produced to indicate delivery result. 7 Triggered by poll() or flush(). """ 8 if err is not None: 9 print('Message delivery failed: {}'.format(err)) 10 else: 11 print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition())) 12 13 if __name__ == '__main__': 14 producer_conf = { 15 "bootstrap.servers": '192.168.0.3:9092,192.168.0.4:9092,192.168.0.5:9092', 16 "security.protocol": 'sasl_plaintext', 17 'sasl.kerberos.service.name': 'kafka', 18 'sasl.kerberos.keytab': '/opt/user.keytab', 19 'sasl.kerberos.principal': 'kafkauser', 20 } 21 p = Producer(producer_conf) 22 23 p.poll(0) 24 p.produce('testTopic', 'confluent kafka test'.encode('utf-8'), 25 callback=delivery_report) 26 27 p.flush() 28 print('done')
If the normal production on the message is configured, using GSSAPI only need to configure security.protocol and the path to the keytab other authentication parameters such as user name and password configurations in different authentication mechanisms, more configuration parameters reference documentation: https: //github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
Details librdkafka SASL configuration process Certified Reference: https: //github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka