Monitor kafka topic, DingTalk alarm

1. The birth of demand

A few days ago, our department of the company needed to demonstrate an application that relied on Kafka data, but Kafka data was delivered from other departments.

Due to some reasons, the data cannot be provided, and the demonstration of our department is also very problematic, so I want to do a simple kafka topic monitoring, find out in time when there is no data, and communicate with other departments

The reason is recorded here, because the bandwidth of the computer room is only 500M, and other departments are doing some video recording work, resulting in full bandwidth, and the producer of Kafka production data cannot be sent to it.

2. kafka monitoring

There are many solutions for kafka detection, but because we use it in a test environment and pay attention to a lightweight one, we just need to write a small program for monitoring.

I have never done kafka monitoring, but I have used Offset Explorer. You can see the total number of messages in Offset. Continuing this idea, as long as this value changes, it proves that it is being delivered again. this is the idea

2.1 Add dependencies

Create a Springboot project, this is not difficult, not to mention

Put all the dependencies here, mainly kafka and the DingTalk sdk that will be used later

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.22</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>alibaba-dingtalk-service-sdk</artifactId>
    <version>2.0.0</version>
</dependency>

2.2 Look at the configuration file

application.yml

spring:
  kafka:
    bootstrap-servers: kafka地址
    consumer:
      auto-offset-reset: earliest
      group-id: test-xiangcai
dingding:
  webHook: https://oapi.dingtalk.com/robot/send?access_token=xxx
  topics: topic1,topic2
  sign: sign

2.3 Write a class to get topic data

@Component
public class KafkaTools {
    private static KafkaTools _this;
    @Autowired
    private ConsumerFactory<Long, String> consumerFactory;
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
    Consumer<Long, String> consumer;
    @PostConstruct
    public void init() {
        _this = this;
        _this.consumer = consumerFactory.createConsumer();
    }
    public static Long getTopicCount(String topic) {
        List<PartitionInfo> partitionInfos = _this.kafkaTemplate.partitionsFor(topic);
        long count = 0L;
        for (PartitionInfo str : partitionInfos) {
            TopicPartition topicPartition = new TopicPartition(topic, str.partition());
            long logEndOffset = _this.consumer.endOffsets(List.of(topicPartition)).get(topicPartition);
            _this.consumer.assign(List.of(topicPartition));
            long currentOffset = _this.consumer.position(topicPartition);
            count += (logEndOffset - currentOffset);
            System.err.println( "  topic  " + topic + "  最后的offset is  " + logEndOffset + "    currentOffset  " + currentOffset + "  totalCount  " + (logEndOffset - currentOffset));
        }
        return count;
    }
}

Explain the key code

LOG-END-OFFSET : The offset value of the latest message currently produced by each partition

CURRENT-OFFSET : The displacement value of the latest consumption message of the consumer group

Note: because the internal use of single-machine kafka, and there is only one partition

3. Send to DingTalk

I want to make a reminder. The original plan was to send emails. I didn’t develop the habit of reading emails, so I wanted to send to WeChat or DingTalk. After reading the plan, DingTalk is the easiest way to implement, so I choose to use DingTalk here.

3.1 DingTalk webhook

Create a DingTalk group, click the group settings in the upper right corner, robot, enter the robot setting interface, then click Add Robot, select Custom Robot, and enter the interface.

 

Here you can set the robot's avatar, the robot's name, and also display the address of the webhook, and you can send messages to the group through this url.

For the method of signing the security settings selected here, you can refer to the documentation, and the code will be displayed later

3.2 Dingding sends messages to the group

Look at the code, the message set here is text, and at all people are used

/**
 * 发送消息到钉钉
 *
 * @param topic
 */
private void sendMsgToDingDing(String topic) {

    try {
        String url = makeUrl(dingDingConfig.getWebHook(), dingDingConfig.getSign());
        DingTalkClient client = new DefaultDingTalkClient(url);
        OapiRobotSendRequest request = new OapiRobotSendRequest();
        request.setMsgtype("text");
        OapiRobotSendRequest.Text text = new OapiRobotSendRequest.Text();
        text.setContent("192.168.2.8 kafka  " + topic + "超过2分钟没有数据，检查下");

        OapiRobotSendRequest.At at = new OapiRobotSendRequest.At();
        // isAtAll类型如果不为Boolean，请升级至最新SDK
        at.setIsAtAll(true);
        request.setAt(at);
        request.setText(text);

        client.execute(request);
    } catch (Exception e) {
    }
}

/**
 * 组装url
 *
 * @param url
 * @param secret
 * @return
 * @throws Exception
 */
public static String makeUrl(String url, String secret) throws Exception {
    Long timestamp = System.currentTimeMillis();

    String stringToSign = timestamp + "\n" + secret;
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret.getBytes("UTF-8"), "HmacSHA256"));
    byte[] signData = mac.doFinal(stringToSign.getBytes("UTF-8"));
    String sign = URLEncoder.encode(new String(Base64.encodeBase64(signData)), "UTF-8");

    return url + String.format("&timestamp=%d&sign=%s", timestamp, sign);
}

look at the effect

 

4. Timer function

The timer uses the springboot schedule directly, and adds @EnableScheduling directly to the application

Create a scheduled job and set it to execute once every 2 minutes. If no data update is found within 2 minutes, a message will be sent to DingTalk to remind you

public static Map<String, Long> countMap = new HashMap<>();

//    @Scheduled(cron = "0/5 * *  * * ? ")   //每5秒执行一次
@Scheduled(cron = "0 0/2 * * * ?") //  2分钟执行一次
public void execute() {
    for (String topic : dingDingConfig.getTopics().split(",")) {
        Long topicCount = KafkaTools.getTopicCount(topic);
        Long orDefault = countMap.getOrDefault(topic, 0L);
        if (orDefault.equals(topicCount)) {
            if (System.currentTimeMillis() - preDingTime > 60 * 60 * 1000L) {
                sendMsgToDingDing(topic);
                preDingTime = System.currentTimeMillis();
            }
        } else {
            countMap.put(topic, topicCount);
        }
    }
    System.err.println("job 执行时间  " + new Date());
}

Here is some processing

The first point is to save the quantity in memory, without using the database, mainly considering the lightweight, in the test environment

The second is the sending of DingTalk messages, to prevent message spewing, here is a limit of 1 hour to send once

5. Package deployment

Look at the dockerfile

#项目所依赖的jdk镜像
FROM openjdk:11-jdk
RUN mkdir -p /app
WORKDIR /app
#设置时区
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
#将maven构建好的jar添加到镜像中，第二个为别名
COPY start.sh start.sh
ADD target/kafkaMonitor.jar kafkaMonitor.jar
#镜像所执行的命令
ENTRYPOINT ["sh","start.sh"]

Look at the startup script

echo "192.168.2.8 kafka.tyjt.com" >> /etc/hosts
java -jar kafkaMonitor.jar
echo 'finish start!!'

It is very simple to start directly. The main feature of this program is a portable

5. Summary

This is just a small requirement, without using some heavyweight components, to solve the problem as quickly as possible.

Give someone a rose, leave a lingering fragrance in your hand, thank you for your likes, attention, and comments.