Read the article: Jingdong orders multidimensional virtual business system architecture design
Article URL: https: //mp.weixin.qq.com/s __biz = MzU1MzE2NzIzMg == & mid = 2247486428 & idx = 1 & sn = 382f9d307073839f7900df7168916cf1 & chksm = fbf7bb33cc80322599a586248c4bf92880374dcb8c48249c91b03170230112492b3ec628206e & scene = 21 # wechat_redirect?
Jingdong Order Center located in the virtual aggregation, change orders and status data maintenance. Orders virtual center mainly around the core functions of data movers (Hamal) product operation, Hamal is relying on Beijing-meter open-source project developed MySQL database binlog monitor products, and the high index monitor the conversion process in the premise of ensuring high availability.
binlog: binglog binary log is provided when mysql database open Row mode, the data recorded in the form of binlogEvent occurrence or potential changes occur (open transaction) SQL statements and data, similar to the oracle database archive log can be used to view the databases change history, incremental backup and recovery and database Mysql replication point of time and so on.
Synchronization Listener principle: in short, from the master analog mysql replication process, first forged into slave send COM_REGISTER_SLAVE command to register the client to the master library, so master will send binlogEvent; then send COM_BINLOG_DUMP command, and specify the file and Position binlog information, can be obtained from the Master library binlogEvent binary stream contains detailed data, binlogEvent contains the type of event (DDL, DML, TCL, authorization, etc.) all of the database, database table information, field information and data lines, the remaining work is filtered, parsed agreement deserialize order to get the desired data.
Hamal as the entrance virtual data center orders in the pipeline, its primary task is to ensure accurate consumer database data changes, and therefore must be carefully designed binlog consumer spending records and exception handling follow-up mechanism.
Fast Consumption: Get Hamal using similar mechanisms and ACK TCP sliding window binlogEvent consumption: receiving bulk binlog each recording, the parallel data parsing changes, the acknowledgment message sent JMQ consumption (ACK), as binlogPosition window length to increase the pace. Hamal MQ message by homegrown way to continue to drive the subsequent service order processing data, the follow-up process consists of changing data deduplication, DML filtration, storage, etc., and can also provide personalized data as risk control, marketing, order transaction systems subscription service. So that can decouple binlog spending is to accelerate spending, but also isolate synchronous monitoring services and business logic.
Separate read and write: Hamal acquired order data into the storage medium continues to flow after a triple order model of the virtual center line of: a conventional database as a first heavy Mysql original data is stored, ES, and caching system for indexing and data analysis, in order to achieve separate read and write. Storing the order data, the DAO decoupling module also uses the message queue, the order data is stored in the database, to form homegrown JMQ push order data to the cache system ES and lightweight stored procedures, reduce inter-coupled and multilevel storage We can balance the cluster load.
Multi-level search: as a data pipeline export orders gateway system (GW) provides customizable templates outside data or news data subscriptions, data paging query, search statistics orders and other services, is the key to butt data applications. Gateway system enables a very complete and powerful multi-stage smooth search process, the search timeout when an order or failure immediately jump to the lower level search, the search results to downgrade the countervailing higher data source; if the virtual center line retrieval fails, the search will fall to product line database for final search, retrieval success will make up the order to the anti-storage centers in all levels of the order, failure to retrieve a single number is bound to be wrong; when the virtual order service completely unavailable, the gateway will direct the search to downgrade to the original product production line database pulls order data. Multi-level retrieval program to ensure the best user experience!