Hengfeng Bank Microservice Architecture Optimization Practice

"Weniger aber besser (less and better)" is the design concept advocated by German industrial design master Dieter Rams, and it is also the essence of Jobs' "reduce complexity to simplicity".

Regarding the architecture, is the more complex the better? Judging from major financial cases, on the contrary, simplicity is the last word. This is also the reason why microservices can be popular. The service architectures in the market, such as EJB, SCA, Dubbo, etc., have more complete functions than microservices, but because they are too complex, they are not as deep as microservices. people's hearts.

At present, the microservice architecture has been practically applied in the scope of financial enterprises. With the continuous optimization of the system, enterprises will choose to improve the original foundation or develop more new architecture platforms suitable for enterprise business. This may be the essence of the popularity of simple frameworks. .

This article focuses on the use case of the micro-service architecture in Hengfeng Bank, and interviews Mr. Zeng Guangyao, senior software architect of the Technology Development Department of Hengfeng Bank, and asks him to talk about the benefits that the micro-service architecture has brought to Hengfeng Bank, and how Hengfeng’s role in micro-services What research and development has been done on the service architecture. PS: Mr. Zeng will also give a speech at the APMCon conference held in Beijing on August 10-11. Those who are interested can pay more attention.

InfoQ: Mr. Zeng, you mentioned that the development team of Hengfeng Bank has developed many self-adaptive services because of performance bottlenecks in the traditional technical architecture. Can you introduce some performance bottlenecks in detail here?

Zeng Guangyao: Most of the traditional application service architectures of banks are based on the J2EE system. They are developed and deployed on the familiar application service containers such as Tomcat, JBoss, and WebLogic. The application software implements object containers through the more popular technical frameworks such as Spring/Struts/Hibernate. Management, MVC layering, data object mapping and other basic capabilities. Most of the application service containers are traditional multi-threaded system architecture models, and respond to external requests through service thread pool scheduling. Due to the high cost of operating system thread switching and other reasons, under the traditional multi-threaded architecture, the number of clients that a single node can serve at the same time is limited, and the concurrent service capability cannot be greatly improved. In our actual production environment, the single-node application service Qps is generally within 1000.

By studying the application technology architecture characteristics of advanced foreign websites and comparing with the existing architecture, we can find that the traditional multi-threaded service architecture cannot effectively utilize the hardware advantages of the current CPU multi-core, and has the following disadvantages:

1. In terms of network multiplexing IO technology selection, most application service containers use the select/poll mechanism, which imposes a heavy burden on the operating system kernel and is prone to C10K problems (that is, a single server node client connection cannot exceed 10,000). In the current mobile Internet application scenario, mobile devices often need to establish a full-duplex long connection, such as WebSocket, with the back-end application server to improve customer interaction experience, which increases the number of devices that can be connected to a single server. Require.
2. A client request occupies a service thread during the execution process. The number of request tasks processed by the application service container at the same time depends on the size of the thread pool. We often need to configure a larger thread pool (one CPU logical core corresponds to more than 10 threads) to achieve more Large concurrent processing capacity. However, in high concurrency scenarios, the larger the number of threads, the larger the operating system's thread switching management burden, which reduces the effective utilization of hardware resources, increases service response time, and reduces server load capacity.
3. The operation of sharing resources between threads requires the help of a lock mechanism, and in high concurrency scenarios, lock conflicts will intensify, thereby affecting the execution efficiency of business logic and the efficiency of thread resource usage.
4. Only synchronous (blocking) IO mechanism can be used in the process of client request execution. In commercial applications, server threads spend most of their time on IO waiting, such as local file reading and writing or network service waiting.
5. The traditional application service container is essentially a single application. It is difficult to program highly fault-tolerant distributed parallel processing. It is difficult to split business logic into multiple parallel processing sub-processes. It cannot effectively take advantage of the hardware advantages of multi-node and multi-core X86 server clusters. The more complex, the slower the response time to the client.

In summary, we believe that the traditional service architecture is difficult to effectively utilize CPU computing resources and realize parallel processing of business logic due to factors such as network multiplexing IO mechanism, thread scheduling, lock resource competition, and synchronous IO, and the overall service carrying capacity Effective, but cannot meet the technical capability requirements of high concurrency and low latency in mobile internet application scenarios.

InfoQ: Briefly introduce the business characteristics of Hengfeng Bank? What are the special specifications for the IT architecture of the entire company?

Zeng Guangyao: Hengfeng Bank is one of the 12 national joint-stock commercial banks. It operates a complete range of financial businesses. In recent years, its business has developed steadily and rapidly. Hengfeng Bank is committed to being "a disseminator of knowledge and technology, a builder of channels and platforms, and a provider of comprehensive financial solutions". Proposed the business strategy of "four-wheel drive, two wings flying together" ("four wheels" refers to corporate finance, financial market, retail finance, mobile financial business, "two wings" refers to investment banking and asset management business), relying on financial cloud Platform and big data platform, realize the four major financial innovation business models including leading finance, platform finance, family finance, O2O finance, build a digital bank, transaction bank, and bank bank, and strive to provide customers and the society with the most efficient and best experience. Comprehensive financial services.

The development strategy of Hengfeng Bank is how to better play the role of financial intermediary, integrate financial service resources and capabilities and combine them with social and economic activities and public life scenarios, so as to realize the innovation of business models with multi-party participation and cooperation in the society. Hengfeng Bank breaks through the limitations of physical outlets, and designs products and business processes in a targeted manner to meet the different needs of subdivided customer groups. Based on digital operations, it gives full play to the advantages of mobile Internet to provide customers with convenient, personalized, Intelligent scene-based innovative financial services. Only in this way can we survive and develop in the fiercely competitive financial market, and we can make use of our strengths and avoid our weaknesses to achieve corner overtaking.

The general requirement of the business development in the bank for the construction of IT systems is to "build a digitally driven smart bank", which is reflected in the following points from the perspective of IT architecture planning and implementation:

1. Make full use of the advantages of cloud computing technology, build a flexible and efficient software service infrastructure that supports flexible deployment, support the application requirements of mobile Internet scenarios with high concurrency and low latency, support the shift of the focus of banking business from offline to online, and meet the needs of customers through various The needs of mobile devices and self-service machines for a good service experience.
2. Build a digital operation system based on big data technology, realize the automatic acquisition, processing and deep mining of massive data, and use data intelligence technology to build the "smart business brain" of Hengfeng Bank to realize intelligent branch service, intelligent risk control system, intelligent Customer acquisition and product personalized recommendation, real-time business analysis and decision support. This also means that for a simple online customer transaction, what happens in the background will become more and more complicated. In addition to the traditional transaction accounting processing, it may also involve the collection of customer online operation behavior and the transaction anti-fraud engine. Rule judgment and pattern calculation, personalized product recommendations and business reminders after transaction completion, and geographic location-based service push of life scenarios; at the same time, real-time customer business views are automatically updated and processed in the background, and different business departments are customized according to their own management perspectives The real-time business analytics dashboard is also recalculated and refreshed.
3. Adhere to the bottom line of safe and reliable operation of the banking system, and provide stable and reliable service quality while actively realizing business innovation. For example, in response to specific transaction peaks such as the "Double Eleven" shopping carnival, it is possible to maintain a stable output of service capabilities; the impact of local program defects can be effectively isolated and automatically repaired, and global service availability problems are unlikely to occur.

InfoQ: What considerations did Hengfeng Bank have in the early stage of designing its microservice architecture, and what potential pits need to be avoided?

Zeng Guangyao: Enterprise application software architecture has experienced Client/Server mode characterized by TCP/IP network communication programming, distributed component object architecture characterized by IDL interface description language (represented by DCOM and CORBA), WSDL/WebService/ESB as the The characteristic SOA service-oriented architecture has evolved to the microservice architecture characterized by Reactive/Containerization/DevOps. Every change of application software architecture claims to solve previous problems and simplify application development, but for most programmers, it seems that they are painfully coerced into the torrent of software evolution and have to adapt to mainstream technology. Architectural changes, and little knowledge in the process often lead to unforeseen problems.

From my personal point of view, the main reasons for the changes in application software architecture in the past 20 years are as follows:

1. Simplifies and shields complex underlying network communication details, enhancing the manageability and location transparency of network services.
2. Split more and more complex real-world applications into smaller software components, constantly weaken the coupling between components, reduce the development difficulty of components, and optimize their combination into the realization form of a complete application.
3. Deploy the decoupled software components as distributed lightweight component services, enhance the autonomy and fault tolerance of component services, and better achieve distributed parallel processing capabilities.

The reason why the microservice architecture is adopted in Hengfeng Bank is mainly because of the complexity of the current application and the flexible deployment requirements of application services. By splitting a large monolithic application system into multiple loosely coupled microservice components that can run independently, it can be divided into smaller teams for independent development according to business and technical functions, reducing the impact of code changes on the overall application, and reducing the impact of code changes on different applications. The application components of the system can flexibly configure hardware resources on demand.

Common misunderstandings that are easy to fall into are:

1. Thinking that the application is just a different running container or infrastructure, everyone still develops and deploys software in the form of a monolithic application.
2. It is not good to grasp the principle of granularity division of microservices. If the granularity is too finely divided, there will be too many microservices, which will increase the cost of interface calls across the network, reduce the response speed, and increase the cost of operation, maintenance and deployment and the difficulty of resource allocation. Isolate the impact between business applications and allocate more hardware resources to software components with performance bottlenecks.

Considering that we need to adopt the Reactive microservice architecture to solve the problems existing in the traditional architecture, we still cut the microservices from the two dimensions of technology and business:

1. Use asynchronous programming mode to package various network service-based microservice components, such as database services, FTP services, MQ services, etc. These services are limited to calls within the application and are generally not open to the outside world. This approach can simplify the complex interactive control details of network service calls, and provide the possibility for pressure load management of back-end network services.
2. For complex iterative algorithms or memory computing tasks that consume a lot of CPU, distributed parallel processing can be better supported after microservices. Algorithms such as Monte Carlo simulation can split the search space and send them to different nodes for actual operations.
3. For software modules with independent business capabilities, the interface is relatively stable or can be designed to be relatively broad (for example, in the form of variable parameter lists), and multiple microservice-oriented business components can be combined into more complex business function services.

InfoQ: What was the reason for ultimately choosing Akka as the underlying software framework for microservices? What functional features of Akka are you more concerned about?

Zeng Guangyao: Akka is a mature solution with high performance, high fault tolerance, and support for elastic deployment. The characteristics of the Actor pattern can be well used to implement microservices. Its detailed features are described as follows:

1. High performance for Akka message processing. A single node can process 50 million messages per second.
2. Low resource requirements, each GB of memory can hold 2.5 million Actor entity objects.
3. Akka's maturity. It is the underlying architecture software of the famous Spark big data platform. It has also been adopted by many large enterprises such as BBC, Amazon, Cisco, eBay, and Groupon, and won the JAX Innovation Development Technology Award in 2015.
4. Based on the powerful Scala language, it has high coding efficiency and can be called with other JVM language class libraries.
5. The Actor concurrent programming model better shields the underlying communication details and thread task scheduling details, making programming easier; based on the message passing mechanism to achieve service collaboration and data sharing, eliminating resource lock competition.
6. The Gossip cluster communication protocol derived from the Amazon Dynamo system has passed the long-term technical verification of the Amazon cloud service platform and has high reliability. It supports Cluster Sharding, Cluster Singleton, Distributed Publish Subscribe, Distributed Data and other cluster application scenarios.
7. Supervisor mode for parent and child actor objects. The parent role object can catch the exception of the child role object and automatically execute different restart strategies, which greatly improves the fault tolerance and service availability of the system, and reduces the difficulty of writing high-availability code.
8. Through the combination of software router (router) and execution dispatcher (dispatcher), a variety of Actor concurrency and load balancing algorithm strategies are implemented to build software elastic deployment capabilities. Among them, the router supports task allocation algorithms such as round robin, random scheduling, broadcast, and consistent hashing; the execution distributor builds an independent thread pool to isolate different types of Actors. The impact of CPU resource contention.

InfoQ: It is understood that Hengfeng self-developed Skyline, a microservice architecture platform based on Akka, to develop related components. Then, what is the effect of finally realizing those requirements? What role does Skyline play within Hengfeng?

Zeng Guangyao: Application development on microservice software frameworks such as Akka is difficult to implement, mainly manifested in the following three points:

1. Limited by the language itself's low level of support for distributed parallel processing, most software development based on microservices or asynchronous IO frameworks needs to pay attention to the interaction details of asynchronous communication, and it is easy to fall into the programming trap of nested asynchronous callbacks, which does not meet most of the requirements. The programmer's habit of sequential programming thinking, development and debugging are very difficult.
2. The basic platform layer lacks the adaptive adjustment ability to cope with peak pressure, and it is easy to transmit the access pressure to the more fragile basic software facilities such as databases, resulting in the overall system collapse.
3. The lack of a differentiated management mechanism for different service qualities at runtime limits the dynamic deployment capability in the cloud computing environment.

In response to the above technical difficulties, we have developed the Skyline microservice architecture platform on the basis of Akka. For the above three types of problems, our technical solutions are:

1. Independently designed and implemented Zebra, a new programming language that supports distributed parallel computing, implements multiple parallel processing paradigms, and supports asynchronous parallel calls of microservices.
2. Componentization of microservices, realizing service component containers, flexible creation and destruction of service component instances, queuing of service requests, time-out cleaning, overload blocking, protecting the more vulnerable service facilities at the backend, and providing stable service quality.
3. The service component container supports dynamic configuration items such as the number of concurrent instances of different microservices, the use of thread resource pools, the length of task queues, the timeout period, and the deployment of physical nodes, which can be dynamically adjusted at runtime to support differentiated service quality.

The final effect achieved by the Skyline platform is:

1. Achieve high concurrency and low latency service capabilities. Zebra distributed parallel service language supports asynchronous parallel service calls through enhanced syntax, improves program running speed and reduces response delay time through data parallelism, instruction parallelism, and Pipeline; Zeroutine coroutine scheduling framework reduces thread switching costs and builds complete Non-blocking program operation mechanism, effectively improve service throughput.
2. Provide stable and reliable service output. The service component container technology realizes the on-demand adaptive adjustment of microservice instances and the queue cache, timeout cleaning and overload blocking mechanism under peak pressure; the component container isolates the impact of program defects and realizes rapid restart after abnormal crash; and through Zebra language The contract-based programming paradigm improves code quality and isolates the impact of bugs in different modules and application systems.
3. Better elastic deployment capabilities. The dynamically configurable service component deployment mechanism supports multiple cluster deployment modes; it implements flexible service deployment strategies from multiple dimensions such as concurrent processing capability, maximum throughput per unit time, and thread resources; it combines with Docker container cloud technology to realize hardware resource deployment. Flexible expansion.

The Skyline platform solves key problems such as system fault tolerance, cluster elastic deployment, rapid development and testing, integration between application services, data parallel processing, and system overload service availability, and provides a complete and effective solution for high-performance data application service development in mobile internet scenarios. Platform tool support; business logic codes written based on Zebra language can effectively shield the technical details of underlying services and speed up the development efficiency of distributed parallel data processing application services. Hengfeng Bank has developed nearly 30 applications and public services based on the Skyline platform in various business fields such as customer marketing and risk management. The QPS value of a single node has reached more than 70,000, and the overall hardware cost of application services has been reduced to 1/2 of the original. 5-1/10, the development cost of high-quality distributed software services is reduced to 1/2, and the procurement cost of commercial middleware software can be saved.

InfoQ: Please introduce how to integrate with the big data software ecosystem under the microservice architecture, and how to quickly develop real-time innovative applications?

Zeng Guangyao: The Skyline platform is mainly integrated and integrated with the big data software ecosystem through the following aspects:

1. Develop related network service adapters. Package the client code of the big data software we can use (such as Kafka, Hbase, Spark, etc.) into an adapter component that can dynamically configure parameters, shielding complex interface interaction details, exception handling and other matters, and this component can be deployed by the container And dynamically managed, asynchronously invoked in Zebra scripting language, and supports multiple parallel processing modes to accelerate applications.
2. A distributed real-time message bus is implemented. It integrates various message facilities such as MQ and Kafka through the service proxy mechanism. Compared with other technical solutions, it provides a unified publish-subscribe programming primitive, which is easy to expand and integrate various message facilities, and can achieve consistent instant message communication between client and client, client and server, and multiple servers. Development; supports pluggable message encoding and decoding algorithms, supports Pub-Sub, RPC, and OneWay interface calling methods, and can directly call Zebra functions on the Web side through WebSocket to achieve distributed parallel data processing.
3. Skyline distributed real-time message bus combines Flume and Kafka to create a complete real-time stream processing platform. Develop data probes for operating systems, databases and various network services, and connect to multiple system data sources; develop management tools to support data collection, formatting, data filtering, aggregate computing, data completion, data storage, and data flow control Dynamic configuration and automatic deployment of other processes; it can dynamically embed business logic processing code written in Zebra language to improve the parallel processing efficiency of back-end services. Compared with other stream processing solutions, business programming is simpler, high concurrency and non-blocking business processing can be realized, multi-system dynamic integration is supported, and the advantages of distributed hardware resources can be fully utilized.
4. Develop a real-time intelligent decision engine for business teams. Business teams can configure business knowledge bases and business rule routing in customer marketing, risk management and other fields through visual tools, support complex rule-driven business logic processing, and realize real-time push of marketing opportunities and real-time early warning of potential risks.

Taking the credit card transaction monitoring application as an example, the implementation steps based on the Skyline platform are as follows:

1. On the stream processing management platform, define the processing rules for real-time collection and dynamic analysis of APM network packet logs, and automatically deploy the agent to the relevant application service nodes; flume collect parses the packet data sent by the agent into structured data according to predefined rules, and transfers it to Kafka's transaction information topic.
2. The stream processing component that executes the Zebra business processing code subscribes to the transaction information topic, and can receive the relevant transaction data in time and store it in the real-time stream cache database asynchronously; the transaction data then triggers the rule inference process of the real-time intelligent decision-making engine, and outputs the classification of transaction risks. data, and forwarded to the Credit Card Transaction Risk Data topic.
3. The subject of credit card transaction risk data is subscribed by the stream business component of risk disposal. According to the risk classification level, it can be determined whether to freeze the card or add a new verification link.
4. Most of the whole process can be realized through the data collection and processing analysis configuration of the real-time stream processing platform, the knowledge base and rule routing configuration of the intelligent decision-making engine, and less Zebra business processing code. Real-time innovative application development on the big data platform will be faster.

InfoQ: What performance optimization and performance tuning measures has Hengfeng Bank made for the microservice architecture? How to improve its high availability performance?

Zeng Guangyao: Hengfeng Bank designed the following performance optimization mechanism when building the Skyline microservice architecture platform:

1. Skyline's coroutine architecture and asynchronous service invocation mechanism, the business logic processing components written by Zebra can achieve non-blocking operation in the container with the number of threads close to the logical core, completely avoiding the cost of thread switching.
2. Network services are packaged through adapter components and deployed separately in Docker, enabling differentiated concurrency control of tasks with different access costs, maximizing the throughput of traditional back-end service architectures, effectively isolating them from business logic tasks, and avoiding the execution of different types of code modules Speed ​​mismatch affects the overall high concurrent service capability of the system.
3. Skyline's distributed real-time message bus enables long-term connections between various devices and back-end services, reducing network latency.
4. The various task parallelization mechanisms provided by Zebra language can improve the parallelism of a single task, make full use of distributed hardware resources, and greatly reduce the response time of tasks.
5. The Skyline platform is based on Actor's component programming mode, which eliminates resource lock competition and improves the execution efficiency of concurrent tasks.
6. Most of the application tuning can be solved by dynamic adjustment of the operation strategy of the Skyline platform component container, which has the convenience of operation and maintenance.

High availability is mainly reflected in the following aspects:

1. Docker containerized deployments enable failover and fast restarts on hardware failures.
2. The component container implements task cache queuing, timeout cleaning, and overload blocking, which can effectively buffer peak pressure and achieve stable and balanced service output; strategic management based on exception handling can effectively isolate program defects and achieve microsecond-level response to abnormal service components. Quick recovery.
3. The key components involved in network services and the distributed real-time message bus have implemented the mechanism of heartbeat activity detection and automatic connection re-establishment, which improves the high availability of network services.
4. Zebra language supports contract programming mechanism, which can detect business logic defects in advance in the development and testing stage, and effectively block the propagation of abnormal program defects in different software components at runtime.
5. The enhanced Raft protocol service component can build cross-center component services, realize multi-center runtime disaster recovery and fault tolerance, and build a never-stop service quality.

InfoQ: Finally, a question of experience, you have 22 years of experience in banking IT system development, can you talk about the stages of change that banking system development has gone through? What characteristics does the current banking system need to have? How to combine with the hottest AI technology and FinTech technology?

Zeng Guangyao: The development and evolution of the IT system of commercial banks basically follows the various development stages of software technology:

1. In the early 1990s, in the host-terminal era, the programming language was C/COBOL. The main IT system of a bank is the core accounting processing system, which is built on a centralized mainframe (small or medium-sized computers are used for large banks, and PC servers are used for small banks). The terminal network system realizes the centralization of business in the same city and the universal deposit and exchange. At that time, a system was promoted, and a hard disk was put in the bag and went out.
2. The mid-to-late 1990s, the era of client/server architecture. The construction of the bank IT system began to propose the concept of a customer-centric comprehensive business system, and began to implement a regionally centralized core banking system. The prefecture and city front-end computer systems were connected to the outlet terminal equipment, and were connected to the provincial bank account processing host through the DDN dedicated line. The communication middleware or through the TCP/IP protocol realizes the message communication and transaction consistency mechanism between the front-end computer and the accounting host. At this time, most of the bank core systems were developed based on Unix systems and C language, and commercial bank online banking systems began to appear, trying to use JAVA language, HTML, and Javascript to develop applications.
3. In the early to mid-2000s, the three-tier architecture era of client/application server/database server separation. Joint-stock banks took the lead in realizing the centralization of core projects in the whole bank, realizing the universal deposit and exchange of the whole bank's business; the application system construction of commercial banks was subdivided, and business statements, credit management, bill systems, etc. were separated from the core accounting system functions, forming a specialization Application; BI and data warehouse technologies have been introduced, and new systems such as CRM system, customer call center system, business decision-making system, and financial management system have begun to be constructed. New systems have begun to adopt J2EE architecture, and JAVA language has gradually replaced C language as the mainstream language for application development. , and some applications use Microsoft's .NET technology.
4. From the late 2000s to the early 2010s, the era of SOA architecture. In response to the fierce market competition, banks began to focus on business model changes and management model changes. The leading joint-stock banks have put forward new concepts and new requirements such as process reengineering and product factories. The SOA architecture conforms to the needs of bank process reengineering to a certain extent, and is also conducive to commercial banks' integration of channels and product systems and the rapid development of innovative business products. roll out. At the same time, the mobile Internet has gradually emerged, and the main channel of customer transactions has gradually shifted to the newly launched mobile banking system. Banks have begun to attach importance to customer service experience, attach importance to the construction of risk management systems, and begin to try data mining and machine learning technologies to extract data value and improve customers. Marketing and risk management levels.

In the mid-2010s, commercial banks faced the pressure of interest rate liberalization and the opening of domestic financial markets to the outside world. The rise of Internet finance also exacerbated the loss of bank customers and funds. Faced with new challenges, commercial banks are also increasing investment in innovation in the field of technology, humbly learning from Internet companies, and constantly embracing new technologies, as reflected in the following:

1. Introduce cloud computing technology to build enterprise private cloud, including OpenStack and Docker container technology, to achieve elastic management of hardware resources.
2. Try to use the micro-service architecture to develop innovative applications, including new systems in distributed core systems, real-time risk warning, real-time marketing applications, etc., hoping to reduce the operational risk of single applications, and more reasonably allocate hardware resources to different business services in the cloud environment. .
3. Build a big data platform and develop innovative data applications and services. The popularity of big data technology has effectively improved the massive data processing capabilities and business modeling capabilities of enterprises. Real-time stream processing technology has improved banks' customer perception capabilities and risk early warning capabilities, and has accelerated business response.
4. Artificial intelligence technology is beginning to gain attention in banks. Machine learning technologies based on big data platforms are becoming more and more popular, accelerating the speed of business modeling in the field of customer marketing and risk management; technologies such as text analysis, cognitive computing, and knowledge graphs have improved the value extraction and development of text data and social network data by banks. Business application level; the application of rule inference engine technology makes intelligent decision-making in complex business scenarios possible.
5. New technologies such as blockchain and VR are in the pre-research and incubation stage in commercial banks. In the process of waiting for the technological system to mature and the implementation cost to drop, they are looking for suitable business scenarios to combine.

{{o.name}}

{{m.name}}