Kafka combat: cluster SSL encryption authentication and configuration (latest version kafka-2.7.0)
Preface overview
- JavaSSL authentication
SSL (Secure Socket Layer), and its successor, Transport Layer Security (TLS), is a security protocol that provides security and data integrity for network communications. TLS and SSL encrypt network connections at the transport layer.
- Kerberos authentication + ACL authentication
Kerberos is a network authentication protocol designed to provide strong authentication services for client/server applications through a key system. ACL is an authentication measure based on Kerberos. Generally, Kerberos authentication is sufficient.
SSL certificate generation
Apache Kafka allows clients to connect via SSL. SSL is disabled by default, but can be enabled if desired.
It can be done using the Java keytool tool. Keytool is a Java data certificate management tool. Keytool stores the key (key) and certificate (certificates) in a file called keystore. The keystore contains two kinds of data:
1). Key entity - secret key or private key and paired public key (using asymmetric encryption)
2). Trusted certificate entities (trusted certificate entries) - only contain the public key
Description of keytool related instructions:
| name | illustrate |
|---|---|
| -alias | Alias, can be customized, here is called kafka240 |
| -keystore | Specify the name of the key store (a certificate store like a database, there can be many certificates, the file cacerts comes with jre, you can also use other file names, if there is no such file name, it will create such a file) |
| -store pass | Specifies the password for the keystore |
| -keypass | Specifies the password for the alias entry |
| -list | Display certificate information in the keystore |
| -export | Export the certificate specified by the alias to a file |
| -file | parameter specifies the filename to export to the file |
| -import | Import the signed digital certificate into the keystore |
| -keypasswd | Modify the password of the specified entry in the keystore |
| -dname | Specifies the certificate owner information. Among them, CN=first and last name/domain name, OU=organization unit name, O=organization name, L=city or region name, ST=state or province name, C=two-letter country code of the unit |
| - come on | Algorithm for the specified key |
| -validity | Specifies how many days the created certificate is valid for |
| -keysize | specify key length |
- Each broker node of the Kafka cluster generates SSL keys and certificates (per broker section execution)
After each node is executed once, each machine in the cluster has a public-private key pair and a certificate that identifies the machine. Note that all broker nodes must execute this command.
keytool -keystore server.keystore.jks -alias kafka240 -validity 365 -genkey
When executing the command, enter first and last name, where you need to enter your hostname, making sure that the common name (CN) exactly matches the fully qualified domain name (FQDN) of the server.
The client compares the CN with the DNS domain name to ensure that it is indeed connecting to the desired server, and not a malicious one.
- Generate a CA certification certificate (in order to ensure the security of the entire certificate, it is necessary to use the CA to guarantee the signature of the certificate)
Although the certificate was generated in the first step, the certificate is unsigned, meaning an attacker can impersonate any machine by creating the same certificate. A certification authority (CA) is responsible for issuing certificates.
The certification body is like the government that issues passports, and the government stamps each passport, making it difficult for the passport to be counterfeited. Additionally, the government verifies the stamp to ensure that this passport is genuine.
Similarly, CA-signed certificates cryptographically ensure that signed certificates are computationally difficult to forge. So as long as the CA is a truly trustworthy authority, clients can have a high degree of assurance that they are connecting to the real machine.
openssl req -new -x509 -keyout ca-key -out ca-cert -days 36
This command can be executed randomly on any broker node, and it only needs to be executed once. After the execution, two files cat-key and ca-cert are generated, and these two files are copied to all broker nodes, which will be used later.
- Create a client trust certificate through the CA certificate (executed by each broker node)
keytool -keystore client.truststore.jks -alias CAKafka240 -import -file ca-cert
- Create a server-side trust certificate through the CA certificate (executed by each broker node)
keytool -keystore server.truststore.jks -alias CAKafka240 -import -file ca-cert
The following is to sign the certificate
- Export the certificate server-side certificate cert-file from the key store (executed by each broker node)
keytool -keystore server.keystore.jks -alias kafka240 -certreq -file cert-file
- Use CA to sign the server-side certificate (executed by each broker node)
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:123456
- Import the CA certificate to the server-side keystore (executed by each broker node)
keytool -keystore server.keystore.jks -alias CAKafka240 -import -file ca-cert
- Import the signed server certificate to the server keystore (executed by each broker node)
keytool -keystore server.keystore.jks -alias kafka240 -import -file cert-signed
Kafka cluster configuration
listeners=SSL://host.name:portssl.keystore.location=/var/private/ssl/server.keystore.jksssl.keystore.password=test1234
ssl.key.password=test1234
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=test1234ssl.client.auth=required
security.inter.broker.protocol=SSL
Client connection configuration
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
test case
kafka-console-producer.sh --broker-list localhost:9093 --topic test --producer.config client-ssl.properties
kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic test --consumer.config client-ssl.properties