Kafka Practice (15): Didi Open Source Kafka Management and Control Platform Logi-KafkaManager Research

Didi has open sourced its Kafka monitoring and control platform Logi-KafkaManager. It has used kafka-manager, kafka-eagle, and kafka-mirrorkaker before. If there are tools that can integrate the advantages of these tools, it is for the production environment The management, monitoring, and resource allocation of the Kafka cluster are very good, so while studying the source code of Kafka, study the source code and usage of Logi-KafkaManager. Didi provides the experience address: http://117.51.150.133:8080/kafka , Account admin/admin .

Reference materials:

github address

product description

Characteristic description

One, source code interpretation

1. Front-end debugging environment

Github cloning is slower and gitee is fast. It adopts a front-end and back-end separation architecture (springboot+reactJS+Typescript). The code contains several modules common, console, core, dao, extends, task, web, among which the web has the start of the MainApplication project. Class, others are dependent. The console module is based on the front-end interface of recat+typescript (the technology stack selection is still very advanced), and the source code of the front-end and back-end runs are viewed locally. Here, the console is run separately in VScode;

# react跟vue一样基于node，所以npm相关依赖引入和配置启动
npm config set registry https://registry.npm.taobao.org 
npm config list #查看npm当前配置
npm install
# 启动react项目
npm start

Start the console front-end module:

console module running

pom.xml needs to comment out the reference to the console front-end module:

2. Post-debugging environment

Depends on Maven 3.5+ (back-end packaging), node v12+ (front-end packaging), Java 8+ (required for operating environment), MySQL 5.7 (data storage), node is not needed because it is placed in vscode, create kafka_manager library in mysql, and Run the sql initialization statement and modify the mysql configuration in springboot at the same time;

mysql --default-character-set=utf8 -uroot -p123456 -P3306 -D kafka_manager < create_mysql_table.sql

Configure MainApplication.java of the web module as the main application class to start;

2021-01-25 19:33:22.642  INFO 18000 --- [           main] c.x.kafka.manager.web.MainApplication    : MainApplication started

Because it is running locally, the proxy target of the console module needs to be modified:

    proxy: {
      '/api/v1/': {
          target: 'http://127.0.0.1:8080',
        //target: 'http://10.179.37.199:8008',
        // target: 'http://99.11.45.164:8888',
        changeOrigin: true,
      }

Above, the debugging environment based on front-end and back-end separation is run independently locally; you can see that the front-end reads the Kafka cluster configuration in the mysql library;

3. Back-end module architecture

Refer to the official application architecture diagram to analyze the code architecture;

+----------------------------+
| Tables_in_kafka_manager    |
+----------------------------+
| account                    |
| app                        |
| authority                  |
| broker                     |
| broker_metrics             |
| cluster                    |
| cluster_metrics            |
| cluster_tasks              |
| config                     |
| controller                 |
| gateway_config             |
| heartbeat                  |
| kafka_acl                  |
| kafka_bill                 |
| kafka_file                 |
| kafka_user                 |
| logical_cluster            |
| monitor_rule               |
| operate_record             |
| reassign_task              |
| region                     |
| topic                      |
| topic_app_metrics          |
| topic_connections          |
| topic_expired              |
| topic_metrics              |
| topic_report               |
| topic_request_time_metrics |
| topic_statistics           |
| topic_throttled_metrics    |
| work_order                 |
+----------------------------+
31 rows in set (0.01 sec)