Reconciliation processing of payment system design: Reproduced

Hill let the soil, which gives it great; Hohai not the small streams, it can on its depth; no king but Zhongshu, it can clear its virtue. ------ Qin Reese "Jianzhuke Shu"

Foreword

We can say that reconciliation is the most difficult thing for the payment system. Each transaction, records must be done each participant can match, no deviation. Working reconciliation system is found records that are different, i.e., rolling account; then manually or automatically manner, to resolve these differences, i.e. flat account.

Electricity supplier system, every transaction, in all relevant subjects should be able to get on the side:

Trading entities, if the sponsor is an individual, you must be able to find a deal from a personal transaction history. But most people do not retain the electronic records, it is generally possible to provide a bill or download transaction records, allowing users themselves to go.
Counterparty, usually a merchant. Settlement process with the merchant side user side, but also provides only statements.
Trading channel side, which is the focus of reconciliation, first, to verify the transaction flow, the second is to verify the transaction commission, after all, hire people to do the settlement channel.
That what records need to be reconciled? At present mainly two: one is the transaction record; a record is refundable. Here's an example to process transactions, refunds records can be treated similarly.

This article is reproduced to the "Phoenix old bear." The payment of interest can check out his blog. http://doc.cocolian.cn/

First, the reconciliation process flow

In general, the reconciliation process involves the following steps: download channels on the bill, local transaction preparation, rolling accounts, balancing the books.

1.1 channels statements download

Banks, third-party payment, UnionPay, will provide basic statement download function. But there are a few bits of work or can not bank too in place, provided only billing inquiries background, the bill does not provide for download. For developers, here are a few pit:

1. Different statements format. Text, XML, csv have. In order to be able to unify the follow-up process, after the download is complete bill needs to be standardized.

2. Different way to download, HTTP, HTTPS, FTP, there. Download the program needs to be handled in accordance with the protocol channel.

3. Download time varies, usually after 1:00, to 12:00 to use, too. If the data is not taken in a predetermined time, it is noted retry reading.

4. Poor stability. FTP server problems that are common. Channel side solution is often the restart. So retry mechanism is necessary.

Look at the case of third-party payment of the bill:

channel	Reconciliation cycle	Bill provide a way	Bill file format
Alipay	2:10 every day	HTTPS	XML
Alipay refund	Daily 3:10	HTTPS	XML
100 Po	7:00 every day	FTP	TXT
100 Po refund	7:00 every day	FTP	TXT
Micro-channel pay	10:30	HTTPS	TXT
Micro letter Refund	10:30	HTTPS	TXT

Bank reconciliations are directly connected

bank	Reconciliation form	Reconciliation cycle	Play money cycle
Bank	Interface merchant reconciliation system	Reconciliation Day	Day junction (T + 1)
CCB	interface	Reconciliation Day	Day junction (T + 1)
ICBC	Online banking login manually download	Reconciliation Day	Day junction (T + 1)
Shanghai Pudong Development	Log credit card self-service platform to obtain reconciliation file, debit card provides reconciliation file through the interface form	Reconciliation Day	Day junction (T + 1)
ABC	Bank reconciliation file timed push	Reconciliation Day	Day junction (T + 0)
Bank of China	Bank reconciliation file timed push	Reconciliation Day	Day junction (T + 1)
Merchants Bank	Bank reconciliation file timed push	Reconciliation Day	Day junction (T + 1)

1.2 channels statements standardization

We look to find examples, such as micro-letter statement, he was csv format, include the following information:

1. Trade Time: This is the side of the micro-channel payment completed. This time will be a trap.

2. Public account ID, merchant number, sub-merchant number, device number: This information needs to be done to verify and ensure their own list, do not let the micro letter to Wang family also gave the list sent me;

3. Micro-channel order number, order number Merchant: These two are single core. The former is the order number of the generation side of the micro-channel, micro-channel with a payment interface in the return value. But what if not receive this return value, it may be in the local record on the air. The latter is the letter we sent to the micro order number, usually to do with this on-one basis. Both sides will have a data value.

4. User ID, transaction type, transaction status, the paying bank, currency, total amount, the amount of red envelopes Enterprise: these is the core of the single field, it must ensure that both are consistent.

5. Trade names, business packets, fees, rates: these are optional verification.
   

The treasure of a statement, a text format, separated by a space. Their home on many simple, only the merchant order number, transaction serial number, transaction time, time of payment, the payer, the transaction amount, transaction type, transaction status of these fields. A treasure for the bill

because the bill format for each channel are different, after receiving the bill, the next step is for the bill to make standardized, so rolling account and follow-up can be unified to deal with. Billing data can be normalized in a file system or database. It depends on the amount of transaction data. The amount of more than one million a day, or use a file system is more appropriate. Database operations are relatively slow, but also a waste of resources. File system-based standardization involves the following:

Csv file format standardized or uniform use of json or xml format. If you are using hadoop or spark to reconcile, use csv is a good choice.

Unified file storage file directory, the file name needs to follow a uniform naming convention.

In order to speed up the processing speed, we use hdfs as a file system, it facilitates the subsequent reconciliation process.

1.3 prepare local transactions

Prepare local transactions, in general the following method:

Consequently do not directly use the raw data. As most systems use mysql, this also means that to do reconciliation on MySQL. Require large amounts of data to find work reconciliation, is bound to affect the online business. When large-scale data, such as more than one million, it is not appropriate.

Of course, another option is to use the standby database to perform a reconciliation, so simple, does not affect the online business. This is typical of the practice space for time.

If a business needs to divide large tables and warehouses to process, reconciliation of data preparation that are not the same. Use sub-library is not realistic, because in accordance with sub-libraries are generally subject id, instead of the channel id, points to the library, so that reconciliation needs to be performed on multiple libraries, but lower efficiency. The library is built on the points table points from the library is also very resource-intensive. In this case, the data needs to be synchronized to a (HDFS) file system, or the NOSQL database.

Since the transaction payment system is the core data, a large number of applications, such as credit, risk control, would require transaction data. These applications demand for the transaction is not exactly the same, in order to improve performance, transaction recording mode will use asynchronous data to be delivered to the consumer. Transactions during storage, delivery of messages to the message system. Consumer monitor the news, once a new message is received, the transaction records from the database query data, get data and updates to the library. Many of such data synchronization on the article, here is not described in detail.

1.4 Rolling account

Rolling accounts in accordance with the customer order number to compare local transactions and transaction channels are the same. From the perspective of the algorithm, it is to calculate the difference between the two arrays. In stand-alone operation, the algorithm can be used a lot, not described in detail here. We recommend to roll mapreduce account, which advantage can be recorded in accordance with the order number will be recorded and local channels offered on the same shuffle to reduce handling, so that you can easily perform data comparison. Rolling posted the largest pit, than the issue of cut points. For example, in order to cut the entire 0:00 minute point, that there is a problem, at 23:59 local transactions initiated, to the side of the channel, may be in the 00:01 deal, this deal becomes the second day of the account. Actual processing, transaction processing in the channel side, are likely to take a few minutes. For cut near points could not confirm the account, do a time window, the data within the time window, the time left to continue processing the next day reconciliation.

1.5 Ping account

Found inconsistent on both sides of the data, it should be how to deal with? When the amount of data, record it, artificial screening on the line. However, if large amounts of data, thousands of daily, manual processing costs would be too high. This is no uniform method of treatment, according to the data in question requires, to be analyzed, and then do automatically. For reconciliation process transactions, mainly the following:

Long section: local non-payment, payment channels has been paid. This is mainly local channels are not correctly received notification issued asynchronous lead. Usually the process is modified to the local state has been paid, and for further processing of the response, such as notifying other party business.
Short paragraph: Local has been paid, but no payment channel record; or no local records, payment channels recorded. The exclusion Kuari factors, in rare cases, do need to know the specific reasons for the treatment.
A discrepancy: Local has been paid, payment channels has been paid, but in different amounts, this requires manual verification.
For reconciliation process refunds, mainly the following:

Local non-refundable payment channels refunded, places subject to payment channels, modify the local state as a refund, and starting subsequent processing.
Local refunded, paid channels refunded, but different amounts, the need for manual verification;
local refunded, but no record of payment channels; or pay channels recorded, but no local. The exclusion Kuari factors, in rare cases, do need to know the specific reasons for the treatment.

Second, reconciliation architecture

Reference architecture based on a system of micro reconciliation services to achieve the following: arch

2.1 pairs bill download

Statements trigger regular daily downloading components, download statements from the payment gateway server. At present, there are HTTP (S) and FTP to download two kinds of statements. The technology selection, HTTP (S) can be realized with HTTP connection pooling and apache httpclient, FTP may be used Apache Commons Net API. No matter which one, you need to set the number of retries and links over time. Retries and interval settings need to be careful, too often retries, the server easily killed; The time interval is too large, will clog the subsequent processing steps. 5 to 10 minutes is a suitable retry interval duration. Link refers to overtime when server problems, connections to obtain data that is not automatically disconnect within the specified time. This can easily be overlooked. We had a problem with your system, FTP is false to restart after the death of side channels, resulting in our client caught, we have been waiting for re-linking. Also, note that some statements to support download page to download.

2.2 pairs conversion bill

Bill bill will be converted to a standard format, providing support for the reconciliation Mapreduce task execution. Each channel of the statements of different formats, need to develop a conversion program, respectively. Conversion program mainly on two operations: parsing the source file into a standard format and output.

2.3 Rolling account MR

As mentioned above, rolling account MapReduce running on Hadoop, transaction number Key, the difference between the check-channel order and local transactions, the difference output record. Finally, the discrepancy record table into the differences.

In short, the reconciliation of work, that is complex nor complicated. Requires care, we have a deep understanding of the business, and select the appropriate architecture.

Thank you for your attention to this article, such as the need for timely receive the latest works which stay in Phoenix, or explore related issues, please scan code concern "Phoenix old bear," the micro-channel public number, number in the message or reply in public, you can as soon as possible, thank you.

This article is reproduced to the "Phoenix old bear." The payment of interest can check out his blog. http://doc.cocolian.cn/