say it up front
The financial platform has undergone two platform architecture evolutions in Murray for more than a year, and encountered various architectural and technical problems in the middle. I have sorted it out today and hope to give you some inspiration.
Architecture Evolution
Overall architecture of the financial platform
business process
The financial platform cooperates with Meili's big data platform and UF NC system to complete the entire financial accounting process.
The big data platform extracts the mysql database data of the business systems connected to the financial platform, such as the company's new core, assets, liabilities, settlement, payment and other business systems, and extracts it into hive through sqoop and further converts it into financial business data. The data is pushed to the financial platform for access In the service database, the batch service of the financial platform pulls up the ledger conversion task for cluster-wide processing, and the financial staff checks the ledger data to create a sub-ledger, and creates voucher data after checking the sub-ledger. The voucher data is pushed to the UFIDA NC service system, and the financial balance management, general ledger management, and related financial report management are realized by the UFIDA NC system.
Financial Platform V1 Architecture
1.0 The application architecture adopts a single mode, and the database architecture adopts a sharding architecture
Background of the project
The application architecture and data architecture of technology must serve the business architecture, and only talking about the technical architecture is just talk about the technology architecture without the business .
In order to meet business needs as soon as possible, the financial team is faced with the problems of professional and complex financial business and huge amount of data. It requires developers to have a good understanding of financial business knowledge, to be familiar with or even proficient in related technologies of massive data processing, and to have very good code performance. Well, the financial platform is built from 0 to 1, and the financial team is also built from 0 to 1. Therefore, in the face of such objective reasons, it is necessary to quickly implement the financial platform, and the flexibility of the financial platform (such as accounting subjects, transaction processing rules, accounting rules) Flexible configuration), scalability (transaction processing rule engine, accounting rule engine, billing engine, and accounting processing engine are abstracted), and accounting processing performance (which can handle massive data) has high requirements, so it is more urgent. The architecture of the database is that the database architecture must be able to withstand such a large amount of data, so the database sharding architecture is adopted. The database architecture is led by my superior, Ah Bing. For the first time, the business history data is 100 million level data checkout, and the current single-day business data volume is about 500W, and the monthly accounting processing data volume is about 5000W.
Why does the application architecture adopt the monolithic model?
Monolithic architecture has many benefits
1. Code sharing
2. Easy to test
3. Easy to deploy
Therefore, if you need to quickly implement a project from 0 to 1, this application architecture solution is the best choice. Although the financial platform adopts a single-mode architecture, the financial platform module design is based on the micro-service architecture design mode, which is used for the later micro-service architecture. The service-oriented architecture has laid a solid foundation.
There is no perfect architecture, only the most suitable architecture. The architecture evolves step by step according to business changes. Otherwise, over-design may be counterproductive.
In the face of such a large amount of data, the financial platform adopts the sharding architecture mode, and the middleware adopts sharding-jdbc. This framework financial platform uses version 1.5, an enhanced database driver based on the client-based implementation of the jdbc specification, which is relatively lightweight At present, the financial platform has carried out a part of customized development for it, and it is now very stable. The current accounting process is a scenario where writing is greater than reading, so this architecture version does not consider read-write separation.
In the architecture version 1.0, the interest accrual service is realized by using the stored procedure logic.
Why use stored procedures
-
Stored procedures can be better modularized.
-
Faster execution, if a SQL needs to be executed in large quantities and repeatedly, the stored procedure is much faster than SQL.
-
Reduce network consumption.
-
The version iteration is relatively fast, and you can only focus on developing SQL, which is more convenient for development and testing.
This database architecture solution is also designed to land quickly.
Financial Platform V2 Architecture
What are the downsides of Architecture 1.0?
Disadvantages of the Monolithic Application Architecture Pattern
1. Not flexible enough
2. It hinders continuous delivery
3. Limited by the technology stack
4. Technical debt, code coupling, affecting the whole body
What are the disadvantages of stored procedures?
1. Execute SQL-level database optimization
2. Running on the mysql server side, if the amount of data is large, the performance of the database server io and cpu will be consumed, and optimization cannot be performed. The server resources are all occupied, which is undoubtedly a disaster for other application services.
Therefore, in Architecture 2.0, we service-oriented interest accrual, replaced all stored procedures, and evolved into a service-oriented architecture model.
Financial Platform V3 Architecture
Why to do service and microservice is to achieve the ultimate goal of high service availability.
The service-oriented architecture in Architecture 2.0 is still relatively coarse-grained, so what are the problems in this architecture mode?
1. Services are still coupled together before, there is no real decoupling
2. The database is still coupled together
3. Not strictly following the single responsibility principle
4. Some services are coupled together and cannot be deployed, upgraded, and replaced independently. The accounting service is a service including ledger, sub-ledgers, and vouchers.
Therefore, based on the current service-oriented architecture, it needs to be further service-oriented to achieve the purpose of micro-service. The technology stack adopted by the financial platform is spring cloud.
What are the benefits of microservices?
1. Easier development, different development division of labor and collaboration, continuous delivery
2. Apps start faster
3. The fault is further isolated. A problem with one microservice will not affect other services, let alone the entire application.
4. In theory, it is not limited to the technology stack, as long as the communication protocol is agreed before the service.
The database adopts read-write separation
Write or sharding-jdbc+mysql database cluster, read using elasticsearch, elasticsearch uses the inverted index mechanism, theoretically more than a hundred times faster than SQL query efficiency, if you want to learn more about elasticsearch, please pay attention to "Tianhe Chat Technology" "WeChat official account, there is a more detailed summary of elasticsearch, and some views on other technologies.
Problems faced after microservices
The biggest problem is the consistency of distributed data. At present, the financial platform adopts strong consistency transactions implemented by TCC. Some other problems are basically solved or customized by the spring cloud ecosystem.
Some thoughts on the integration of industry and finance on the financial platform
At this stage, the financial platform is an afterthought system.
With so many systems and business scenarios connected to the financial platform, data governance requires a long and arduous process, such as scenarios such as settlement, payment, refund, offline settlement, offline repayment and other business scenarios. The data obtained by the platform is not real-time and completely accurate, and may also be a kind of business-neutral data. Therefore, the current business data to the financial platform is connected to the business data through the bridge of the big data platform. Therefore, the financial platform cannot Real-time perception of data on the business side, so to achieve real business-financial integration, decision makers can see financial reports in real time. The real-time and accuracy of financial reports is still unattainable for the time being. This is also what the financial platform plans to achieve in the future. Only by connecting the financial platform and business data in real time, the business system is all online, the financial platform performs real-time bookkeeping, financial reconciliation process automation, and accounting processing process automation, in order to achieve real-time financial reporting, and ultimately achieve business results. purpose of financial integration.
The current processing plan is to extract the business data on T-1 day for financial accounting processing. Therefore, the pressure on the database and the server is very large when the data volume is concentrated together. The server is basically idle at other times in the interval, which is largely a waste of server resources, and this is also where the financial platform architecture needs to be optimized later.
This architectural plan is currently only an idea, and has not yet been planned in detail. It is an idea. It will take a long and arduous process to get to this point.
To achieve this goal requires these things
1. Business data governance and operations are all online.
2. The business data and the financial platform are connected in real time.
3. The accounting process is fully automated.
said the last
The above is a summary of the application of the three versions of the financial platform and the evolution of the database, for reference only.
Companionship is the longest confession of love
Push dry goods for you every day
Identify QR code
follow me