Kafka configuration 2-configure Kafka SASL-PLAIN authentication under Windows

1. Zookeeper configuration
1.1. Modify the Zookeeper configuration file.
In the Zookeeper installation directory conf, open zoo.cfg for editing, and add the following configuration to the file:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

   1.2. Create a new zk_server_jaas.conf
       In the Zookeeper installation directory conf, create a new zk_server_jaas.conf file and add the following content:

Server {
     org.apache.kafka.common.security.plain.PlainLoginModule required
     username="admin"
     password="admin123456"
     user_quber="quber123456"
     user_quber1="quber123456"
     user_quber2="quber123456";
};

Description:
           username and paasword are the authentication between zk clusters. It
           defines three users quber, quber1 and quber2. The password is quber123456, which is used by Kafka clients to access Zookeeper
           :
           quber: for reading and writing quber1: for reading
           quber2 : Used to write

   1.3 and modify zkEnv.cmd
       in the Zookeeper installation directory bin, open zkEnv.cmd for editing, in the file set ZOO_LOG4J_PROP=INFO, the following configuration is added to the next line of CONSOLE:

set SERVER_JVMFLAGS=-Djava.security.auth.login.config=D:/Net_Program/Net_Zookeeper/conf/zk_server_jaas.conf

Note that the above configuration is a slash, not a backslash


   1.4. Import the relevant jars
       . Find the following jars in the Kafka installation directory libs respectively, and copy them to the lib in the Zookeeper installation directory to
           kafka-clients-2.4.0.jar
           lz4-java -1.6.0.jar
           slf4j-api-1.7.28.jar
           slf4j-log4j12-1.7.28.jar
           snappy-java-1.1.7.3.jar

2. Kafka configuration
   2.1, create a new kafka_server_jaas.conf
       in the Kafka installation directory config, Create a new kafka_server_jaas.conf file and add the following content:

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin123456"
    user_admin="admin123456"
    user_quber="quber123456"
    user_quber1="quber123456"
    user_quber2="quber123456";
};

Description:
           In the KafkaServer section, username and password are used by the broker to initialize the connection to other brokers.
           In the above configuration, the admin user
           defines the communication between the brokers and user_userName defines all the client connections connected to the broker and the broker authentication,
           including other brokers User password, user_userName must be configured with the admin user, otherwise an error will be reported.

   2.2. Create a new kafka_client_jaas.conf
       In the Kafka installation directory config, create a new kafka_client_jaas.conf file and add the following content:

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="quber"
    password="quber123456";
};

Note:
           In the KafkaClient section, username and password are the users used by the client to configure the client to connect to the broker.
           In the above configuration, the client uses the quber user to connect to the broker

   2.3. Modify server.properties
       in the Kafka installation directory config, and open the server. Edit the properties and add the following configuration to the file:

listeners=SASL_PLAINTEXT://192.168.2.200:9092
advertised.listeners=SASL_PLAINTEXT://192.168.2.200:9092
# 使用的认证协议
security.inter.broker.protocol=SASL_PLAINTEXT
# SASL机制
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
# 完成身份验证的类
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
# 如果没有找到ACL（访问控制列表）配置，则允许任何操作。
#allow.everyone.if.no.acl.found=true
super.users=User:admin

Note:
           The IP address behind listeners is the internal network IP
           . The IP address behind advertised.listeners is the external network IP

   2.4. Modify consumer.properties and producer.properties
       in the Kafka installation directory config, open consumer.properties and producer.properties for editing, Add the following configuration to the file:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

Modify group.id=TestGroup1 in consumer.propertie. If there are multiple consumer clients, ensure that the group.id is not duplicated, otherwise they cannot consume messages at the same time.

   2.5. Modify kafka-server-start.bat
       in the Kafka installation directory bin\windows, Open kafka-server-start.bat for editing, and add the following configuration in the file under the SetLocal keyword:

set KAFKA_OPTS=-Djava.security.auth.login.config=D:/Net_Program/Net_Kafka/config/kafka_server_jaas.conf

Note that the above configuration is a slash, not a backslash.


   2.6, modify kafka-console-consumer.bat and kafka-console-producer.bat
       in the Kafka installation directory bin\windows, open kafka-console-consumer.bat and kafka-console -producer.bat to edit, add the following configuration to the next line of the SetLocal keyword in the file:

set KAFKA_OPTS=-Djava.security.auth.login.config=D:/Net_Program/Net_Kafka/config/kafka_client_jaas.conf

Note that the above configuration is a slash, not a backslash


3. Start the service and configuration
   3.1, start the Zookeeper and Kafka services
       separately Run CMD as an administrator (open 2 windows to start the Zookeeper and Kafka services respectively), and then enter:
       zkServer
       kafka-server-start D:\Net_Program\Net_Kafka\config\server.properties

   3.2. Setting user permissions
       Here is the configuration of the three users mentioned above (quber[read and write], quber1[read] and quber2[ Write]) read, write and group permissions

       3.2.1, set the read and write permissions of user quber
           to run CMD as an administrator, and then enter:

kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --add --allow-principal User:quber --operation Read --operation Write --topic TestTopic1

kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --add --allow-principal User:quber --operation Read --group TestGroup1

Description:
               The first line of the above command is to set the "read" and "write" permissions
               for the user quber. The second line of the command is to set the permissions for the user quber in the group TestGroup1. If the second command is not set, it will consume data When there is no access to TestGroup1 group permissions

       3.2.2, set the read permissions of quber1
           to run CMD as an administrator, and then enter:

kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --add --allow-principal User:quber1 --operation Read --topic TestTopic1

kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --add --allow-principal User:quber1 --operation Read --group TestGroup1

Note:
               The first line of the above command is to set the "read" permission
               for user quber1. The second line of command is to set the permission for user quber1 in the group TestGroup1. If the second command is not set, there is no access when consuming data

       3.2.3 of the group permission of TestGroup1 , set the write permission of quber2
           to run CMD as an administrator, and then enter:

kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --add --allow-principal User:quber2 --operation Write --topic TestTopic1

Description: The
               above command is to set the "write" permission for user quber2

       3.2.4, query the user operation permission
           Run CMD as an administrator, and then enter:

.\bin\windows\kafka-acls.bat --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --list --topic TestTopic1
.\bin\windows\kafka-acls.bat --authorizer-properties zookeeper.connect=192.168.2.200:2181 --list --topic TestTopic1
或
kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --list
kafka-acls --authorizer-properties zookeeper.connect=192.168.2.200:2181 --list --topic TestTopic1

Description:
               The function of the above command is to list all


       users with the authority of TestTopic1 3.2.5, delete the authority of the user ,
           run CMD as an administrator, and then enter:

.\bin\windows\kafka-acls.bat --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --remove --allow-principal User:quber --operation Read --topic TestTopic1 --force
或
kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=192.168.2.200:2181 --remove --allow-principal User:quber --operation Read --topic TestTopic1 --force

Description:
               The function of the above command is to delete the read permission of user quber in the topic TestTopic1.

       3.2.6. To sum up
           , we have configured SASL to complete, and then we can use the client of each language (such as C#) to read and write Send the message to Kafka.
           If you need to dynamically add other read-write accounts, you only need to add the account and the corresponding password in the file kafka_server_jaas.conf, and then follow the steps 3.2.1~3.2.3 to set the account permissions.

           Note:
               In the customer When consuming data on the client side, if you use different accounts to consume the same topic, you need to set different accounts into different groups. Then the client that starts the consumption cannot consume data. After
               the account is dynamically added and the account permissions are set, you need to restart Kafka Service, otherwise it will not take effect

3.3 Test
3.3.1 Start Kafka's own producer
, run CMD as an administrator, and enter:

.\bin\windows\kafka-console-producer.bat --broker-list PLAINTEXT://192.168.2.200:9092 --topic TestTopic1 --producer.config .\config\producer.properties
或
kafka-console-producer --broker-list PLAINTEXT://192.168.2.200:9092 --topic TestTopic1 --producer.config D:\Net_Program\Net_Kafka\config\producer.properties

       3.3.2. Start the consumer that comes with Kafka
           to run CMD as an administrator, and then enter:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server PLAINTEXT://192.168.2.200:9092 --topic TestTopic1 --from-beginning --consumer.config .\config\consumer.properties
或
kafka-console-consumer --bootstrap-server PLAINTEXT://192.168.2.200:9092 --topic TestTopic1 --from-beginning --consumer.config D:\Net_Program\Net_Kafka\config\consumer.properties

Note:
               After setting in 3.3.1 at this time, the above command can start the consumer normally.

4. Refer to the document
   KAFKA permission configuration SASL/PLAIN authentication: https://www.mscto.com/cloud/262978.html
   belated Dry goods | Kafka permission management combat: https://blog.csdn.net/u013256816/article/details/99619334

Kafka configuration 2-configure Kafka SASL-PLAIN authentication under Windows

Guess you like