[Deploy kafka on Tencent's lightweight application server and read kafka data through flink]

Environmental preparation

After a month of exploration, I finally chose to build a learning environment on Tencent Cloud. At that time, the reason for the choice was that there was a discount for new users (about 150 for 3 years), but now the 1-core 2G configuration is barely enough. It is recommended that subsequent friends choose 2-core 4G configuration.
Since it is a single-node installation, the following resources need to be prepared:
1. jdk1.8
2. zookeeper3.5.9
3. kafka_2.12-3.0.0
link: all resources are integrated here .
Extraction code: pbtw

JDK installation

1. Find jdk to decompress:
   

2. Configure environment variables

    vi /etc/profile
    export JAVA_HOME=/usr/local/soft/jdk1.8.0_171
    export PATH=.:$JAVA_HOME/bin:$PATH
   

   Execute after configuration: source /etc/profile

zookeeper installation

1. Find zookeeper to decompress:
2. Configure environment variables (as above)
3. Enter the conf directory of zookeeper after decompression
4. execute mv zoo_sample.cfg zoo.cfg
5. Modify the following path
   

install kafka

1. Find kafka to decompress:
2. Configure environment variables (as above)
3. After decompression, enter the config directory of kafka: vi server.properties, modify the following:
   
   
   

As above, the required resources have been configured, and the following starts

Common commands

启动（kafka目录下执行）：
zkServer.sh start
bin/kafka-server-start.sh -daemon config/server.properties

停止（kafka目录下执行）：
bin/kafka-server-stop.sh config/server.properties
zkServer.sh stop

创建topic（kafka bin目录下执行）：
kafka-topics.sh --create --topic test --bootstrap-server 10.0.4.2:9092 --partitions 1 --replication-factor 1

查看topic（kafka bin目录下执行）：
kafka-topics.sh --describe --topic test --bootstrap-server 10.0.4.2:9092

创建生产者（kafka bin目录下执行）：
kafka-console-producer.sh --topic test --bootstrap-server 10.0.4.2:9092

创建消费者（kafka bin目录下执行）：
kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server 10.0.4.2:9092

After starting zookeeper and kafka, you can see the following process:

start production and consumer commands to consume data:

The installation on the above server is successful, and then read kafka data through flink

Open the firewall port number

This step is very important, because the cloud server has firewall restrictions enabled by default. If you access a specific ip port number locally, you need to configure it in the firewall rules.
1. Enter the cloud server homepage management page

2. Click the firewall to add rules

3. Add ports 9092 and 2181 respectively

4. Add as follows

add dependencies

我这里用的是flink1.12版本
```xml
<dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>1.12.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>1.12.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.12.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_2.12</artifactId>
            <version>1.12.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.0.0</version>
        </dependency>

code example

import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;

import java.util.Properties;

public class kafkaTest {
    
    
    public static void main(String[] args) throws Exception{
    
    
        //创建上下文
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        test1(env);
        env.execute("kafkaTest");
    }

    public static void test1(StreamExecutionEnvironment env) {
    
    
        Properties prop = new Properties();
        prop.setProperty("bootstrap.servers","云服务器外网ip地址:9092");
        prop.setProperty("group.id","test");
        DataStream<String> stream = env
                .addSource(new FlinkKafkaConsumer<String>("topicname",new SimpleStringSchema(),prop));
        stream.print();
    }

}

Open a producer client on the server and send data into it

The result of the operation is as follows:

The above completes the entire deployment and calling process. There will be a record of stepping on the pit later.