Author | kimmking
Source | CSDN blog, editor | Xi Yan
Selling | CSDN (ID: CSDNnews)
With the development of distributed technology, MQ technology products also have blowouts. At present, in addition to all kinds of commonly used MQ, such as Apache ActiveMQ, Kafka, Pulsar, RocketMQ (both Apache and Alibaba, the headline is also based on RocketMQ), and RabbitMQ (Meituan, car home a lot of use), major manufacturers have Since I researched my own products, Tencent's CMQ and TubeMQ, JDQ from JD, QMQ from Qunar, and DDMQ from Didi (based on RocketMQ), many of them are open source. Here talk about the open source TubeMQ this year.
Tencent open source TubeMQ
The official introduction is as follows:
https://github.com/Tencent/TubeMQ/blob/master/docs/tubemq_basic_introduction_cn.md
TubeMQ is a distributed messaging middleware system (MQ) developed by Tencent Big Data in 2013, focusing on high-performance storage and transmission of massive data in the context of big data. After nearly seven trillion years of massive data precipitation, compared with many open source MQ components, TubeMQ has certain advantages in mass practice (stability + performance) and low cost. Recently, we have relevant code and design of open source TubeMQ , More information is being collated.
TubeMQ cluster architecture:
After years of evolution, the TubeMQ cluster is divided into the following five parts:
Portal: Portal part responsible for external interaction and operation and maintenance operations, including API and Web. API connects to the management system outside the cluster. Web is a page encapsulation of daily operation and maintenance functions based on API;
Master: responsible for the control part of the cluster control, this part is composed of one or more Master nodes, Master HA through the heartbeat keep alive between the master nodes, real-time hot backup switching is completed (this is when you use TubeMQ Lib, you need to fill in the corresponding cluster all The reason of the master node address), the master master is responsible for managing the status of the entire cluster, resource scheduling, permission checking, metadata query, etc .;
Broker: Store part responsible for the actual data storage. This part is composed of independent Broker nodes. Each Broker node manages the Topic collection in this node, including the addition, deletion, modification, and checking of Topics, and the messages in Topics. Storage, consumption, aging, partition expansion, offset consumption of data consumption, etc., the external capabilities of the cluster, including the number of topics, throughput, capacity, etc., are completed by horizontally expanding Broker nodes;
Client: The client part responsible for data production and consumption. We provide this part in Lib format. The most used by everyone is the consumer. Compared to before, the consumer now supports Push and Pull data pull modes. Support both order and filtered consumption. For the Pull consumption mode, the service supports resetting the accurate offset through the client to support the business extract-once consumption. At the same time, the consumer end has launched a new cross-cluster switch-free BidConsumer client;
Zookeeper: Responsible for the zk part of the offset storage. This part of the function has been weakened to only the persistent storage of the offset. The module is temporarily retained considering the next multi-node copy function.
Compared with the conventional distributed MQ structure, the broker function is relatively heavy.
Compared to Kafka, TubeMQ's system features:
Pure Java implementation language: TubeMQ is developed in pure Java language, which is convenient for developers to quickly familiarize themselves with the project and problem handling;
Introduce Master coordination node: Compared to Kafka which relies on Zookeeper to complete metadata management and implement HA guarantee, the TubeMQ system uses a self-managed metadata arbitration mechanism. The Master node uses the embedded database BDB to complete the metadata in the cluster The storage, update, and HA eager functions of TubeMQ cluster are responsible for the operation management and configuration management operations of the TubeMQ cluster, and provide external interfaces; through the Master node, the Broker configuration settings, changes, and queries in the TubeMQ cluster implement a complete automatic closed-loop management, which reduces the The complexity of system maintenance;
Server-side consumption load balancing: TubeMQ adopts the service-side load balancing scheme, not the client-side operation, which improves the management and control capabilities of the system and simplifies the implementation of the client, which is more convenient for the upgrade of the balancing algorithm;
System row-level lock operation: Row-level locks are used for concurrent operations with intermediate state in the reading and writing of Broker messages to avoid duplication;
Offset management adjustment: Offset is managed by each Broker alone, ZK is only used for data persistence storage (initial consideration is to completely remove the ZK dependency, and it is temporarily retained in consideration of subsequent function expansion);
Improvement of message reading mechanism: Compared to Kafka's sequential block reading, TubeMQ adopts a random message reading mode, and at the same time increases memory cache reading and writing in order to reduce message delay. For machines with SSD devices, increase message lag Transfer the processing of SSD consumption to solve the problems of throughput decline when the consumption is seriously lagging, the small capacity of SSD disks and the limited number of flashing times, so that it can meet the needs of rapid production and consumption of business (detailed in the following chapters);
Consumer behavior management and control: Supports real-time and dynamic control of consumer behavior accessed by the system through policies, including limiting the current of specific services when the system load is high, suspending consumption, and dynamically adjusting the frequency of data pulling;
Service grading management and control: According to the different needs of system operation and maintenance, business characteristics, and machine load status, the system supports operation and maintenance to dynamically control the consumption behavior of different consumers through strategies, such as whether they have permission to consume, consumption delay grading guarantee, and consumption limit Control, and data pull frequency control, etc .;
System security management and control: According to the different data service needs of the business and the consideration of system operation and maintenance security, the TubeMQ system adds a TLS transport layer encryption channel, authentication and authorization of production and consumption services, and access token management for distributed access control , To meet the needs of business and system operation and maintenance in terms of system security;
Improved resource utilization: Compared with Kafka, TubeMQ uses connection multiplexing mode to reduce connection resource consumption; through logical partition construction, it reduces the system's occupation of file handles, and through server-side filtering mode, reduces network bandwidth resource utilization; By stripping the use of Zookeeper, reduce Zookeeper's strong dependence and bottleneck restrictions;
Client improvement: Based on the convenience of business use, we simplified the client logic to make it the smallest set of functions. We used the reception quality statistical algorithm based on the response message to automatically remove the bad Broker node, based on the first Use connection attempts to avoid sending blockages when sending large amounts of data (for details, see the following chapters).
This piece basically clarifies the characteristics and some of the characteristics of other MQs. In fact, it can be guessed that it has been compared with kafka. Many places have participated in and improved kafka, and they have made a lot of thinking and new management capabilities. Implementation.
TubeMQ client evolution:
The business that has the most contact with TubeMQ is the consumer side. How to adapt to the characteristics of the business and facilitate the use of the business? We have made more improvements in this area:
The data pull mode supports Push and Pull:
Push client: The initial consumer version of TubeMQ only provides consumption in the Push mode. This mode can consume data faster and reduce the pressure on the server, but it also brings a problem. When the business is used, the pull frequency cannot be controlled. It is easy to form a data backlog and data processing cannot come;
Push client with consumption suspension / continuation: After receiving business feedback on whether it is possible to control the push pull action, we added the resumeConsume () / pauseConsume () function pair to allow the business to simulate the waterline control mechanism and compare the status When busy, call pauseConsume () function to stop Lib background data pull. After the state is restored, call resumeConsume () to inform Lib background to continue pulling data;
Pull client: We have added a Pull client in a later version. This client is different from the Push client. It is the business rather than Lib that actively pulls the message and confirms the success of the data processing result. The initiative for data processing is left to the business. After this treatment, although the pressure on the server side has increased, the backlog during business consumption can be greatly alleviated.
Data consumption behavior supports order and filtered consumption: At the beginning of the design of TubeMQ, we considered that different businesses use different topics. In actual operation, we found that many businesses actually report data through the proxy mode, and the data passes through the file ID or table under the topic. To distinguish by ID attribute, the business needs to consume all the data under the topic in order to consume its own data. We support the filtering consumption mode of the specified attribute through the tid field, and place the data filtering on the server to reduce the traffic and the data processing pressure of the client.
Support business extract-once consumption: In order to solve the need for accurate file back when business processes data, the client version provides the function of resetting the precise offset through the client. When the service restarts the system, only the client needs to provide the callback At the point of consumption context, TubeMQ can continue to consume according to the specified precise location. This feature is currently used in real-time computing frameworks such as Flink, relying on Flink to perform extract-once data processing based on the checkpoint mechanism.
Push and pull are the two most basic modes of message processing. Pushing is simpler for server processing. It doesn't matter if it is pushed out. The broker becomes lighter, but it may push too much per unit time, resulting in a backlog on the consumer side and overwhelming the client-side system. Pull means that when you come to fetch data at any time, the broker must maintain its state and generate a backlog, and it also needs to deal with retry strategies and so on. With offset, it means that the message can be traced back at any time, but this may cause duplication. If there is no built-in deduplication, it is not extractly once, but atleast once, the message will be repeated.
Several other mq
Didi's DDMQ:
https://github.com/didi/DDMQ/blob/master/README_CN.md
QMQ of Qunar:
https://github.com/qunarcorp/qmq
Interesting points
What is the difference between TubeMQ and mainstream MQ architectures such as kafka, rocketmq, and pulsar?
The official opinion is:
Kafka is implemented in a sequential write + sequential block read mode. The performance data under a single instance is very strong, but as the number of instances increases, its performance shows an unstable decline. TubeMQ uses a sequential write + random read mode, even at the maximum Under the restriction, the system can still achieve long-term stable inflow of more than 1G, and at the same time, combined with the server-side filtering, the consumption of filtering is very smooth.
Personally have reservations about this, a large number of topics are not suitable for Kafka's design principles (generally we recommend that the number of topics in a single cluster is less than 100, too many small topics cause random reads and writes, but they can be merged, and then differentiate and route messages ), At the same time, if it is changed to an SSD disk, it can also improve throughput and delay, with few thousand topic problems. And the latency of kafka is not like the 250ms compared in the above document, we actually use it between 10-40ms.
TubeMQ took a look, the overall design is a bit like pulsar, mainly because the broker and storage are separated; the message processing mode is a bit close to ActiveMQ.
Several interesting places:
1. TubeMQ does not support multiple copies. In this case, the single machine may still lose data in extreme cases, but multiple copies are the current standard configuration of various distributed message queues. (Look at the commercial version CMQ on Tencent Cloud. .)
2. Consumption load balancing on the server side, this is the case with the earlier version of kafka, and there are many problems
3. Messages are read at random, so it needs to add memory cache and rely on SSD, which is very weird. It is very complicated to add locks for concurrency. ActiveMQ is because the processing of memory is too complicated, resulting in a large amount, which is not good for anyone.
4. Support push and pull at the same time, which is also very interesting. It has something to do with the first one. If you support push, the server must definitely have a state.
5. Support message filtering on the server side. Now the general MQ is client-side filtering, the same is true.
MQ found that up to now, it has experienced three generations, represented by ActiveMQ, Kafka / RocketMQ, and Pulsar. From the trend point of view, it is more and more distributed and tends to support cloud native. Thin and light.
In short, this solution seems to integrate some of the characteristics of traditional and current MQ, but the implementation is very heavy.
There is also a tip. The component names in TubeMQ are a bit confusing. The thing called master is actually a broker. The thing called broker is actually storage (bookie in pulsar).
：）
Original link:
https://blog.csdn.net/KimmKing/article/details/103133789
【END】
Today's benefits
Meet Lu Qi
Also as an important part of "One Million People Learn AI", the 2020 AIProCon Developer Ten Thousand Conference will be broadcast live online from July 3rd to 4th, allowing developers to learn about the cutting-edge technology of AI in one stop Research, core technology and application, and practical experience of enterprise cases, and can also participate in exciting and diverse developer salons and programming projects online. Participating in the forward-looking series of activities and online live broadcast interaction, not only can communicate with tens of thousands of developers, but also have the opportunity to win exclusive gifts for live broadcasts, and even join the technology giant.
Tickets are limited to the big show! Starting today, click to read the original registration "2020 AI Developer Ten Thousand Conference", use the coupon code "AIP211", you can get a free online conference ticket worth 299 yuan. Limited to 100 sheets, first come first served! Come and use your finger to get membership for free!
Click to read the original text and go directly to the official website of the conference.
More exciting recommendations
☞One -stop killer AI development platform is here! Say goodbye to switching scattered modeling tools
☞The idea of intelligent transportation caused by the traffic jam of Beijing Fourth Ring Road
☞ Please, do not ask me what is the heap!
☞The idea of intelligent transportation caused by the traffic jam of Beijing Fourth Ring Road
Every "watching" you order, I take it seriously
