TubeMQ Tencent distributed messaging middleware system in 2013 from the research, focused service high-performance storage and transmission of large data scene of massive data, after nearly seven years of massive data trillions of precipitation, average daily access to more than 25 trillion. Compared with open source MQ components to many stars, TubeMQ in the mass practice (stability + Performance) and has better core aspects of low-cost advantage.
TubeMQ Apache Foundation donated
September 12, Apache Software Foundation on the occasion of the 20th anniversary, Tencent announced TubeMQ open source in ApacheCon. TubeMQ start Apache Foundation plans to donate the process.
TubeMQ System Features
1. pure Java language
Tube MQ using pure Java language developers, for developers to quickly become familiar with the project and deal with the problem;
2. Introduction of Master coordinator node
Zookeeper compared to Kafka relies on complete management of metadata and implement different HA guarantee, Tube MQ system uses self-manage metadata of arbitration, Master complete the storage nodes in the cluster by using metadata embedded database BDB, update and HA eagerly function, responsible for tube MQ cluster operation control and configuration management operations, provide external interfaces; by Master node, tube MQ cluster Broker configuration settings, changes and inquiries to achieve a complete automated closed-loop management, reduce system maintenance complexity;
3. The server-side load balancing consumption
Tube MQ uses a load balancing server side programs, the client side rather than the operating end, to enhance the ability to control the system while simplifying client implementation, upgrading easier balancing algorithm;
4. The system of row-level locks operation
For an intermediate state concurrent operation Broker message read line-lock used, to avoid duplication;
5. Offset managing adjustment
Offset independently managed by the respective Broker, ZK only for storing persistent data (ZK removed completely dependent initially considered, taking into account the subsequent expansion function on escrow);
6. Improved message read mechanism
Tube MQ message using the random access mode, and in order to reduce the delay message buffer added to read and write memory, SSD device with respect to the machine, increasing the processing SSD message transfer lag consumption, solving throughput drops and the time of consumption lags behind small SSD disk capacity, disk brush limited number of issues, to meet the needs of business fast production and consumption;
7. Consumer Behavior control
Support consumer behavior control system access through dynamic real-time strategy, including a high system load current limiting for a specific service, pause consumption, dynamic adjustment of frequency and other data pulled;
8. grading service control
For system operation and maintenance, operational characteristics, different needs of the machine load status, operation and maintenance support system to dynamically control strategies by different consumer behavior, such as whether there is consumer rights, consumer classification to ensure the delay, limiting consumption control, and data pulling frequency control;
9. The system security control
Depending on the needs of business data services, system operation and maintenance as well as security considerations, Tube MQ system adds TLS pipeline transport layer encryption, authentication production and consumption of services, authorization, and access control for distributed access token management to meet business and system maintenance on demand system security;
10. enhance resource utilization improvements
Compared to Kafka, Tube MQ connection using multiplexing mode, connections to reduce resource consumption; logical partition is configured to reduce the occupancy of system file handles, filtered through a server-side model, reduce network bandwidth usage; by peeling of the Zookeeper use, and to reduce the strong dependence of Zookeeper bottlenecks;
11. Improved client
Based on the convenience to business use, we simplifies the client logic, so do the minimum feature set, we use the Broker node automatically bound by the bad quality of statistical algorithms based on the received response message, based on first use for connection blocked attempts to avoid sending large amount of data transmitted.
Welcome to the open source collaboration
We warmly welcome you to participate in the open source collaborative TubeMQ and each module of the project in the past, if you encounter problems when using TubeMQ project, or have an idea for improving TubeMQ project, welcome your issue!
TubeMQ officially open
Github Open Source Address:
https://github.com/Tencent/TubeMQ
(Click to read the original text at the end direct access)
Please give TubeMQ a Star!
We welcome your issue and PR!
TubeMQ domestic mirror address: https://git.code.tencent.com/Tencent_Open_Source/TubeMQ
(Login to access public project)
Tencent worker bees source system to provide a complete open-source developers, the latest domestic image Tencent open source project
