Cloud platform queue service-Qbus practice

 Fan Quanqing  360 Cloud Computing  2

Heroine declaration

The editor of this article starts from the introduction, architecture, and usage scenarios of the Qbus service. It does not involve the specific implementation principles of the bottom layer of the queue. It only talks about the process of uploading the Qbus service to the private cloud. I hope to bring some value to everyone with the same needs. thing. Let's follow the author and study together.

PS: Rich first-line technologies and diversified forms of expression are all in the " 360 Cloud Computing ", please pay attention!

1

Service Introduction

 

As a cloud platform queue service, Qbus serves various business lines of the company. The current daily traffic is PB+ , topic 1W .

Qbus to kafka components as the core and the surrounding supporting services, including 日志收集, 各种语言的sdk,  持久化存储到hdfs, 业务定制监控and other services.

2

scenes to be used

  There are a wide range of scenarios, including common scenarios such as log systems, management services, big data, and stream processing. The main role , ,  and so on.   服务解耦 消息通讯 流量削峰  

3

Service architecture

 

4

Service access method

 

There are three main types of read-write clients:

• Log collection;

• sdk write;

• Reading and writing of other components or platforms (such as flink, hadoop);

For the first two, you can submit a work order directly on the cloud platform, and it will be automatically approved. The third type requires manual communication. We provide multi-version clusters to adapt to more business platforms.

5

Service concrete realization

Among them, I combine the realization of the entire service. Talk about what we mainly did and the problems we encountered. This article only talks about solutions. No specific details are involved.

Log collection components:

The log collectionlogstash components have been customized, and the version will be launched filebeatimmediately after R&D.

General users apply for log collection on the cloud platform topic, fill in the machine, and the collection path, an automatic work order will be generated, and the command system will automatically go to the business machine to install logstash. Of course, it also includes daily path additions, deletions, modifications, and inspections. These are implemented by providing an interface for the cloud platform.

Customization features include :

• Unified configuration management, automatic update;

• Collect progress reports;

• Service self-test;

• Add the machine name to the log header, and merge special logs;

• bug modification;

The configuration is stored in the cloud platform configuration center.

The status is shown in the figure:

6

SDK

 

The SDK is self-developed in the group. Kafka-bridge is currently open source. Everyone is welcome to use.

Custom features include

• Use VIP instead of broker_list;

• More powerful error handling, skip directly when there is an unavailable copy;

• Related parameter optimization;

7

Continuously save to HDFS

This is a supporting service around the Qbus queue, providing the function of persisting to HDFS.

HDFS we directly use the platform of the brother department.

The service structure is as follows:

As shown in the figure, the following are mainly realized:

• For consumer component usage flume, we made it containerized;

• The relevant configuration is stored in the cloud platform configuration center and supports dynamic changes;

• We have added a dynamic adjustment tool to the periphery to monitor consumption status in real time and automatically add or subtract copies;

• It also saves the historical state and displays it uniformly, so that it is well-documented;

In this way, the user only needs to click on or off on the page, which is very convenient.

8

Monitoring service 

As the basis of the overall service, the monitoring service provides data support for the overall service, including the dynamic adjustment tools of the upper layer, which rely heavily on monitoring.

Of course, businesses also have permission to view the traffic of their topics on the cloud platform. 

We mainly use the monitoring service Prometheus.

As shown in the figure:

The following modifications were made to it:

• Divide into multiple processes according to business level and do HA;

• Unified back-end storage (influxdb cluster);

• nginx is added as a reverse proxy before prometheus;

• Increase authorization verification;

This provides a stable monitoring data interface for the upper layer.

Summary & Outlook

Now in Qbus services relying on the cloud platform has done work orders automatic, unless kafka cluster failures require manual processing.

However, as more and more businesses put services on the container platform, horizontal expansion has become very fast, frequent, and exception handling may not be reasonable, which also brings more challenges to basic services.

Basic services need to be more robust to cope with all this.

This includes:

• More rapid expansion and migration capabilities;

• Richer business isolation;

• More powerful client compatibility;

Having said that, I look forward pulsarto joining the new version of qbus.