Alipay System Architecture Overview
Typical Handling Defaults
Funds Processing Platform
Financial Accounting
payment clearing
Accounting Center
trade
flexible affairs
Alipay's open source distributed message middleware - Metamorphosis (MetaQ)
Metamorphosis (MetaQ) is a high-performance, high-availability, and scalable distributed messaging middleware, similar to LinkedIn's Kafka. Scenarios such as sequential message, broadcast and log data transmission have been widely used in Taobao and Alipay, and are now open source.
Metamorphosis is a Java message middleware open sourced by Taobao. Regarding messaging middleware, you should have heard about the JMS specification, as well as some open source implementations such as ActiveMQ and HornetQ. Metamorphosis is one of them.
The origin of Metamorphosis is that I started from the study of linkedin's open source MQ - now transferred to apache's kafka. This is a very uniquely designed MQ system. It uses the pull mechanism instead of the general MQ push model. It makes extensive use of Zookeeper does service discovery and offset storage. I appreciate and agree with its design concept. I strongly recommend that you read its design document. Generally speaking, the design of metamorphosis is completely consistent with it. But why do you still need meta?
Briefly summarize the reasons why I re-write the meta:
1. Kafka is written in scala, I am not familiar with scala, and the development of the whole kafka community is too slow.
2. There are some functions that kakfa has not implemented, but we need: transactions, multiple offset storage, high availability solutions (HA), etc.
3. Some features of Meta relative to kafka:
- Text protocol design, very transparent, supports protocols like memcached stats to monitor brokers
- Pure Java implementation, from communication to storage, from client to server are re-implemented.
- Provides transaction support, including local transactions and XA distributed transactions
- Support HA replication, including asynchronous replication and synchronous replication, to ensure message reliability
- Support for sending messages asynchronously
- Failed to consume message, support local recovery
- A variety of offset storage support, database, disk, zookeeper, can be customized to support group commit, improve data reliability and throughput.
- Support message broadcast mode
- A series of supporting projects: python client, spout for twitter storm, tail4j, etc.
Therefore, the improvement of meta compared to kafka is huge. Meta has been widely used in both Taobao and Alipay. Now Alipay routes 12 billion messages per day through meta, and Taobao also has hundreds of millions of messages per day.
Meta is suitable for applications
- Log transmission, high-throughput log transmission is the strength of kafka;
- Message broadcast function, such as broadcast cache configuration invalidation;
- Sequential synchronization of data, such as mysql binlog replication;
- Message routing in a distributed environment (broker, producer, and consumer are all clusters), scenarios that have extremely high requirements on order and reliability;
- Other functions used as general MQ.
Author: Xue Ji
Source: Mobile Payment Network (WeChat Public Account: mpaypass)