360 internal message queue system Qbus introduction

360 internal message queue system Qbus introduction

Infrastructure Group 360 Cloud Computing

Heroine declaration

As the basic components for processing big data for every Internet company, a series of message queuing systems such as Kafka and rabbitMQ are more and more favored by server-side programmers. In order to maintain data durability, scalability and high availability, the team In view of the company's internal status quo and business characteristics, based on kafka, a message queue system Qbus that meets the internal characteristics of 360 has been deeply customized. Today, the young master will give you this dry goods sharing about the message queue, hoping to help you.
PS: Rich first-line technology and diversified forms of expression are all in the "HULK first-line technology talk", please pay attention!

Introduction to Qbus

origin:

• LinkedIn's distributed message queue kafka for log processing;
• Real-time processing of user behavior (login, browse, click, share, like) and system operation logs (CPU, memory, disk, network, system and process status) is required;
• Traditional queues are too heavy and performance cannot keep up;

Features:

• Messages are persistent and stored on disk;
• High throughput
• The client maintains the consumption status;
• Distributed, producers, consumers, and servers can all be distributed;
• Support subscription;
• Push/pull method;

Messaging is
used as a common messaging system (synchronous and asynchronous messaging).
Log collection
collects business logs for real-time analysis, stores and other
data synchronization
uses subscription mode, synchronizes data to various computer rooms,
monitors services,
monitors service access, and prevents users from accessing the website. Restricted crawling data,
activity flow tracking, etc.
Combine Storm and Hadoop to consume data on Storm, storm processing results are stored in QBus, and data can be asynchronously copied to hadoop

Compared with other queues, compared with gearman, JMS has the following advantages:

• Messages can be persisted, so there is no worry about message loss;
• Distributed, smooth expansion of the server, no need to perceive the business
• No single point (multiple computer room guarantee);
• Support subscription, one message, multiple receiving (support cross-computer room structure, convenient for parallel processing);
• High throughput
• Support message batch processing;
• Simple to use

Qbus principle introduction

Schematic diagram:

meta information:

1. Cluster broker information, including its ip:port
2. The storage location of topic
3. Consumption location information of partition
4. All consumer information
5. Coordinator on duty
6. Agent configuration and survival status information

The path of a message from production to consumption:

broker cluster:

Fully peer-to-peer between multiple brokers (smooth expansion is possible)

Broker storage:

• Each partition corresponds to a logical log, composed of multiple segments
• The new message is appended to the end of the last segment file
• Consumers can go back to the location where the message exists according to the offset to re-consume

The key to Broker's high performance:

• Use file system cache, user space does not cache messages
• zero-copy (saving 60-70% time than read-send)
• End-to-end batch compression
• Sequential read and write disk is very efficient
• Do not care about the status of the message (whether it is consumed, where it is consumed, etc.)

performance:

Support php, C/C++, java version SDK, the trend chart of performance and message size is as follows:
(2.40GHz CPU single core environment, broker in the same computer room)

Code writing java:

Producer: (asynchronous)

Consumer: (callback mechanism)

Producer: (synchronous, asynchronous)

Consumer: (callback mechanism)

The improvement of QBus over kafka

1. Increase the type of reliable transmission request, the broker will confirm the message;
2. Add agent nodes for asynchronous message collection and log collection;
3. Increase the coordinator node, independent consumption distribution logic, solve the thrilling group effect, and reduce the cost of multi-language development;
4. Improve php sdk according to QBus system design, and rewrite C/C++, java SDK;
5. Add a message self-service management platform to facilitate users to manage messages;