Kafka_2.12-3.1.0 cluster environment construction

1. Environmental description

---

The current Kafka version still requires the use of zookeeper to store clusters, brokers, consumers and other related meta-information. The current version, namely version 2_12_3.1.0, has built-in zookeeper. : If you choose to use the 故针对zookeeper环境有两种built
1. 使用内置的zookeeper
-in zookeeper , you need to modify the config in the Kafka installation directory. The zookeeper.properties file in the directory
2. 使用外部的zookeeper
requires the zookeeper cluster to be installed in advance.

在本案例中，我们采用了外部的zookeeper, that is, a three-node cluster zookeeper environment was built, which was used as the metadata repository of Kafka2_12_3.1.0 version.

1. The zookeeper environment configuration is as follows :

node	installation path	dataDir path
hadoop002	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp
hadoop003	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp
hadoop004	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp

2. The kafka environment configuration is as follows :

node	installation path	Log path
hadoop002	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/
hadoop003	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/
hadoop004	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/

2. Installation steps

---

1. Upload the Kafka installation package, unzip and install it

   tar -zvxf kafka_2.12-3.1.0.tgz -C /training
   
        
        
         
         
   
        
        
         
         
   

2. To modify server.propertiesthe file, you only need to modify the following items:

   broker.id=0
   log.dirs=/training/kafka-cluster/kafka-logs
   zookeeper.connect=hadoop002:2181,hadoop003:2181,hadoop004:2181
   group.initial.rebalance.delay.ms=3
   
        
        
         
         
   
        
        
         
         
   

   其他配置默认即可

3. Distribute the entire Kafka installation directory to other nodes

    scp -r /training/kafka-cluster root@hadoop003:/training/
    scp -r /training/kafka-cluster root@hadoop004:/training/
   
        
        
         
         
   
        
        
         
         
   

4. Go to the other two nodes and modify the server. properties to broker.id: 1和2,
   note :broker.id 不得重复

5. Start the Kafka cluster

   1. Start the zookeeper cluster first (start by yourself)
   2. Start the Kafka cluster on hadoop002, hadoop003, and hadoop004 in sequence, and execute the following command:

      kafka-server-start.sh server.properties &
      
             
             
              
              
      
             
             
              
              
      

      注意：该命令是相对路径下执行且以后台进程方式启动Kafka
   3. Verification: Execute: jpscommand on three nodes and you will see that the process name is:Kafka

1. Environmental description

---

The current Kafka version still requires the use of zookeeper to store clusters, brokers, consumers and other related meta-information. The current version, namely version 2_12_3.1.0, has built-in zookeeper. : If you choose to use the 故针对zookeeper环境有两种built
1. 使用内置的zookeeper
-in zookeeper , you need to modify the config in the Kafka installation directory. The zookeeper.properties file in the directory
2. 使用外部的zookeeper
requires the zookeeper cluster to be installed in advance.

在本案例中，我们采用了外部的zookeeper, that is, a three-node cluster zookeeper environment was built, which was used as the metadata repository of Kafka2_12_3.1.0 version.

1. The zookeeper environment configuration is as follows :

node	installation path	dataDir path
hadoop002	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp
hadoop003	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp
hadoop004	/training/zookeeper-3.4.5	/training/zookeeper-3.4.5/tmp

2. The kafka environment configuration is as follows :

node	installation path	Log path
hadoop002	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/
hadoop003	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/
hadoop004	/training/kafka-cluster/	/training/kafka-cluster/kafka-logs/

2. Installation steps

---

1. Upload the Kafka installation package, unzip and install it

   tar -zvxf kafka_2.12-3.1.0.tgz -C /training
   
        
        
       
       
   
        
        
       
       
   

2. To modify server.propertiesthe file, you only need to modify the following items:

   broker.id=0
   log.dirs=/training/kafka-cluster/kafka-logs
   zookeeper.connect=hadoop002:2181,hadoop003:2181,hadoop004:2181
   group.initial.rebalance.delay.ms=3
   
        
        
       
       
   
        
        
       
       
   

   其他配置默认即可

3. Distribute the entire Kafka installation directory to other nodes

    scp -r /training/kafka-cluster root@hadoop003:/training/
    scp -r /training/kafka-cluster root@hadoop004:/training/
   
        
        
       
       
   
        
        
       
       
   

4. Go to the other two nodes and modify the server. properties to broker.id: 1和2,
   note :broker.id 不得重复

5. Start the Kafka cluster

   1. Start the zookeeper cluster first (start by yourself)
   2. Start the Kafka cluster on hadoop002, hadoop003, and hadoop004 in sequence, and execute the following command:

      kafka-server-start.sh server.properties &
      
             
             
            
            
      
             
             
            
            
      

      注意：该命令是相对路径下执行且以后台进程方式启动Kafka
   3. Verification: Execute: jpscommand on three nodes and you will see that the process name is:Kafka