Community sharing｜Guosen Securities' system quality assurance practice based on open source tools

Editor's Note: In the "2023 MeterSphere Open Source Continuous Testing Platform City Encounter·Shenzhen Station" event held on June 17, 2023, Lin Ying, senior testing manager of Guosen Securities, shared a keynote speech entitled "System Quality Assurance Practice Based on Open Source Tools". The following content is organized based on this speech.

Guosen Securities Co., Ltd. ("Guosen Securities" for short) is a national large-scale comprehensive securities company with 58 branches and 183 sales offices in 117 cities and regions across the country. Guosen Securities has 4 wholly-owned subsidiaries including Guosen Futures Co., Ltd., Guosen Hongsheng Private Equity Fund Management Co., Ltd., Guosen Capital Co., Ltd., and Guosen Securities (Hong Kong) Financial Holdings Co., Ltd., and holds 50% of the shares in Penghua Fund Management Co., Ltd. The business scope of the company and its subsidiaries covers securities brokerage, margin financing and securities lending, consignment sales of financial products, market making transactions, futures brokerage, asset management, investment consulting business, business management service business, etc.

Lin Ying, Senior Test Manager of Guosen Securities

1. The background and challenges of Guosen Securities system testing

1. Background of Guosen Securities System Testing

As a leading local securities company in China in Shenzhen, Guosen Securities has extremely high requirements for system quality, and its requirements for testing quality assurance have also increased accordingly. Generally speaking, information systems in the financial industry usually have the following characteristics:

① Two-state IT system architecture with high technical complexity

Financial information systems generally have "two-state" business systems such as steady state and sensitive state at the same time.

Among them, the steady-state business system mainly refers to the system that carries the core business of securities companies such as user account opening, business processing, securities trading, entrustment cancellation, transaction offer, and unified liquidation. The construction of steady-state business is generally earlier. The steady-state business system design of Guosen Securities adopts a more traditional bus mode, and some of the systems have been running on the Windows operating system for about 20 years;

The sensitive business system is a new business system that carries intelligent stock selection decision analysis and other categories in order to meet the needs of fund business diversification and innovation. Sensitive business was born late, and its system technology architecture is updated compared with steady-state business. Most of it adopts micro-service architecture, and a small part adopts containerized deployment method to better meet the demands of rapid business iteration.

The sensitive system of the securities business is more oriented to the Internet user side. After the user-side business enters from the sensitive system, it reaches the exchange and other places through the long link of the stable core business. Based on the coexistence of "stable" and "sensitive" system conditions, the selection of test products needs to better meet the test requirements of the business system under the "two-state" mode.

② Multi-mode delivery form, with a large number of systems

In addition to the main business system, Guosen Securities also has various support systems such as an operation platform and an information platform. For these support systems, if they match the company's business needs after market research, these mature system products will be introduced through outsourcing; if there are certain customization needs, they will cooperate with manufacturers to build them; in addition, the company sometimes adopts internal and completely self-developed methods for system construction. The sources of these supporting systems are different, and the delivery modes are different, which makes the quality management of the whole system more complicated.

③ The industry is strictly supervised and the fault tolerance rate is extremely low

The information system of the financial industry is directly related to the safety of users' funds. Failure to troubleshoot and resolve system failures in a timely manner can easily lead to asset losses for users during use. Therefore, regulators will have stricter requirements on the company's IT team. Once a problem occurs in the system, it must be dealt with quickly within the specified time, otherwise the team will face serious accountability and punishment.

2. Challenges faced by financial information systems

The financial system of Guosen Securities is mainly divided into important system and general system. These two types of systems have different focuses and need to be maintained in different ways, which means that they will face different challenges in the process of system construction and operation and maintenance.

① Important system

The important system is the system including account opening business, trading business and some clearing business. Most of the important systems are related to important information such as customer funds, and are the key protection objects of the company. The responsible IT team needs to have the sensitivity and speed of bug troubleshooting, so it will invest more resources to strengthen testing, acceptance, and operation and maintenance to ensure its operation quality.

The challenges faced by important system quality assurance mainly include:

■ Verification of business system launch depends on external colleagues

At present, there are 350 components of various types in Guosen Securities, and the application system will be changed about 3,000 times a year. Moreover, the company will cooperate with the Shanghai Stock Exchange and Shenzhen Stock Exchange to conduct some customs clearance tests every week, there will be regular system restarts every two weeks, there will be disaster recovery drills every month, and there will be annual disaster recovery drills every six months. After each customs clearance test or system restart and other activities, it is necessary to conduct an all-round verification of the business system again.

Due to regulatory requirements, the verification work is more dependent on the staff of the sales department to assist. However, the work of the sales department is mainly to serve customers. The staff of the sales department often do not have a deep understanding of the IT system, and cross-departmental communication and collaboration can easily lead to information loss, and some omissions may occur during the system verification process.

■ Business failure reports depend on customers

The business failure report of Guosen Securities follows the process of "customer finds a problem and gives feedback → account manager feedback → first-line emergency personnel report → second-line operation and maintenance personnel feedback → developer handles it". The entire process link of business fault reporting is too long, and the time spent on feedback increases synchronously. Difficulties in backtracking and troubleshooting at the problem site will also affect the repair time. This kind of delay and inefficiency may touch the regulatory red line of the CSRC, and it may also easily lead to further expansion of the scope of the impact, turning small problems into big problems.

To sum up, the risk management and control of important systems is mainly reflected in the process of acceptance testing and continuous operation after acceptance.

② General system

Guosen Securities invests less resources in general systems than in important systems. For general systems, there is usually only one project manager in charge of management, taking care of various tasks such as technical control, requirements design, system testing, and system operation and maintenance.

For general systems, the main challenge is the inability to effectively control the quality of systems provided by manufacturers. Guosen Securities generally obtains general systems through direct outsourcing, but the quality of the products of the manufacturers is often uneven. In the system delivery mode, the requirements design, development coding, system testing and other parts are mainly carried out by suppliers spontaneously, and it is difficult for Party A to intervene and carry out quality control in advance. Only when the system is delivered, an outsourced colleague will conduct a round of purely manual acceptance testing on site. The quality risk of system delivery is high, and the regression verification period after defect repair is relatively long. From the perspective of quality assurance, the risk management and control of the general system must start before the system goes online.

2. Construction ideas of quality management and business monitoring

In view of the problems faced by different types of systems, Guosen Securities has sorted out some ideas for solving the problems, as follows:

1. Implement the "three unifications" standardized manufacturer quality management

The "three unifications" include unified test management, unified standard deliverables, and unified tools. The three are interrelated and promote each other, and can achieve the goal of "simultaneously improving the quality and efficiency of the delivery system". Among them, in the scenario of unified test management, the unification of test tools can be better promoted; unified test tools and corresponding unified test standards can promote the formation of standard deliverables; the formation and quantity growth of standard deliverables will also feed back unified test management. Through such a management idea, Guosen Securities can gradually improve the process management of unifying and standardizing the quality of manufacturers.

2. Actively monitor the business layer to detect system availability failures in advance

In response to the pain points of important systems in the company, in order to achieve quality assurance for important business matters such as online acceptance and customer reimbursement, the IT team of Guosen Securities chose to try the idea of "shifting testing to the right" and moving the testing process to the production environment. In the production environment, the user's business call operation can be automatically simulated to perform real-time business dial-up testing, improve the efficiency of release verification, perceive the availability of the system in advance, and ensure the stable operation of the system.

When the company's business changes, the team can also quickly find faults in the production process or system changes by running interface tests or automated tests in the production environment, buying more time for system troubleshooting and reducing the negative impact on user experience.

3. Open source tools help traditional enterprises quickly build IT infrastructure at low cost

Based on the previous two construction ideas, the IT team of Guosen Securities decided to adopt the "tools first" model, with the help of low-cost open source testing tools, to quickly build IT infrastructure and ensure its operating quality.

The IT team of Guosen Securities has the following considerations for deciding to use open source testing tools instead of purchasing closed source products, customizing or self-developing testing software:

① The cost of open source tools is low

In addition to the lower explicit costs such as product costs, the implicit costs of open source tools such as learning costs and time savings in use are also more advantageous than other testing tools. Because of the wider circulation of open source tools, employees are more likely to have the experience of learning and using the software, and it is easier to learn software operation skills with the help of rich community tutorials, which virtually saves the learning cost of the testing team;

② Short period, quick effect

Existing test tools will be more mature than custom-made or self-developed test tools, and can achieve the effect of "out of the box", which is less difficult to connect to the business system and has a shorter construction period;

③ more advanced technology

Open source testing tools often use mainstream technology stacks, and the technology selection is advanced, which can lower the threshold of software learning and use to a certain extent;

④ Strong product vitality

The users of open source software are more extensive, and the products will continue to iterate with the continuous feedback of a large number of users, and they will have the power of continuous evolution, and their vitality is stronger than that of closed source software, customized software or self-developed software.

3. The implementation of MeterSphere in Guosen Securities

1. Selection of open source tools

After detailed investigation and selection, the IT team of Guosen Securities chose MeterSphere, a one-stop open source continuous testing platform. The most attractive advantages of MeterSphere include:

① The open source community is highly active

MeterSphere has a wide user base, and the open source community is very active. At present, MeterSphere has obtained more than 9,000 Stars and more than 2,000 Forks on GitHub, and has been used in many enterprises;

② Comprehensive functions

As a one-stop open source continuous testing platform, MeterSphere includes almost all functions such as test tracking, interface testing, and performance testing required by Guosen Securities' testing work;

③ Compatible with JMeter

Before the introduction of MeterSphere, the performance testing of Guosen Securities was mainly based on JMeter. There are some non-universal digital protocols in the company's trading system and other steady-state systems, which are also compatible and supported through extension plug-ins developed based on JMeter. Since MeterSphere is also compatible with JMeter, after the introduction of the MeterSphere platform, the docking of the test work between the two can be completed conveniently and quickly;

④ Strong scalability

MeterSphere is compatible with multiple protocols and has a rich plug-in system. This means that MeterSphere, as a piece of software, has sufficient scalability and can better adapt to many internal systems of Guosen Securities;

⑤ Test environment adaptation

MeterSphere supports containerized deployment, which is convenient for the rapid migration of various test environments such as Xinchuang environment and the adaptation with test tools;

⑥ Adopt mainstream technology stack

MeterSphere is mainly written in Java language, using common technology stacks such as Spring Boot (backend) and Vue.js (frontend), and is highly compatible with Guosen Securities’ business systems and IT team members’ skills.

2. Implementation of business monitoring

① All-round coverage of business monitoring

Through the interface automation function of MeterSphere, the IT team of Guosen Securities has achieved 100% coverage of business monitoring in the core business system including trading, wealth management, account opening, home page, market price, information, etc., as well as 100% coverage of the self-built data center and market data cloud computer room, using the "Golden Sun" mobile APP as a pilot. A total of 591 interfaces are covered, and the interface coverage rate reaches 94.5%. Business monitoring maintains uninterrupted operation for 7*15 hours (09:00~23:59, non-stop on holidays).

② Closed-loop implementation of monitoring alarm discovery, push, response, and processing

With the assistance of MeterSphere, Guosen Securities has realized the closed-loop management and control of monitoring and alarm discovery, push, response and processing. In the case that the business monitoring keeps running continuously for 7*15 hours (09:00~23:59), when an error occurs in the business system, the business monitoring will issue a corresponding alarm, and push the alarm to the unified alarm event platform and WeChat of the person in charge, so as to quickly reach the IT staff responsible for problem handling.

At the same time, the platform can classify and classify these alarms, assisting staff to quickly prioritize problems and make processing decisions. For alarming faults, MeterSphere can also give clear information prompts such as equipment room location, interface location, error content, and error time. The IT team also dyed the parameters of business monitoring, so that it is easy to know whether the error submitter is an internal staff or an external user.

When the company's business system changes, a large number of alarms may be triggered in a short period of time. The IT team devised a configurable automatic recovery strategy for alerts. After the system change is completed and the service returns to normal, there is no need to waste manpower to manually check and deal with them one by one. These invalid alarms can be automatically restored to normal.

③ Realize support for private protocols and various non-standard protocols

The business system of Guosen Securities also involves a variety of private protocols and non-standard protocols. The IT team solved this problem through an extensible plug-in mechanism on the MeterSphere platform. During the whole process, according to the three-step model of "depending on the manufacturer-cooperative research-independent development", 4 plug-ins have been expanded, including two plug-ins developed by Feizhiyun's customer success team, a private TPC gateway plug-in jointly developed by the IT team of Guosen Securities and Feizhiyun, and a Huarui speed trading plug-in independently developed by Guosen Securities. In the future, Guosen Securities will independently expand the plug-in system of the MeterSphere platform according to the requirements of the business system and test requirements.

④ Play value in application changes, system restarts and daily business inspections

The business monitoring system built through the MeterSphere platform has played a very high value in Guosen Securities business system changes, restarts and daily business inspections. Within one year after the business monitoring system was launched, the IT team discovered a total of 61 online problems, 72% of which were discovered during routine inspections, and 28% of which were discovered during application changes and daily restarts. As a financial company, Guosen Securities' online failure rate has always been at a relatively low level among similar companies.

3. Business system quality management based on MeterSphere

① Clarify the system quality process management norms

By using the MeterSphere test platform, Guosen Securities has broken the black box of the system supplier, and clarified the whole-process management specification for the quality of the system delivered by the cooperative manufacturer, that is, "shifting" the test management process to the left before the delivery by the manufacturer.

After the system requirements design, Guosen Securities will conduct a preliminary review of the requirements design, and then the system supplier will start the development and coding. After the development and coding are completed, the team will first communicate with the manufacturer to conduct a use case review, allowing the manufacturer to conduct system testing according to the requirements, and require the manufacturer to standardize the management of the output. In the end, when Guosen Securities conducts internal test and acceptance of the business system, it will perform a process audit on this process, and the online deployment of the system can only be opened after passing the audit.

② Unified platform, standardized deliverables, quick acceptance and archiving

Guosen Securities requires system suppliers and Guosen Securities to use a unified test platform that is compatible with data—that is, MeterSphere, a one-stop open source test platform. On the basis of delivering business functions, the manufacturer needs to simultaneously deliver the written functional test cases, interface automation use cases and performance test cases to Guosen Securities, and these test-related delivery contents can be directly imported into the internal environment of Guosen Securities through the MeterSphere platform. In this way, during the maintenance period after system procurement, operations such as system upgrade or service migration can be performed quickly without the intervention of the system supplier, and the existing use cases can be directly run to carry out testing work to achieve seamless system delivery and standardized management of manufacturers and delivery systems.

③ Provide various service capabilities for different projects and roles

The MeterSphere platform provides diversified service capabilities for different projects and roles in the IT team of Guosen Securities.

■ For developers , MeterSphere supports debugging, editing, and playback of multiple interface protocols, and at the same time provides interface mock services in the front-end and back-end separate development modes, and can also realize one-click conversion of interface testing to performance testing, effectively improving the efficiency of software development;

■ For test managers , MeterSphere can systematically manage asset data in the test process, facilitate the storage and reuse of use cases, and can also conduct online review of use cases directly on the MeterSphere platform. In the "Test Management" module of MeterSphere, test tasks can be assigned online and the test progress of the staff can be tracked in real time. For the testing work carried out on the MeterSphere platform, it is also possible to audit the testing process in a more standardized manner, update the audit results in real time and leave traces for the execution actions. Defects found in the audit can be classified and processed to ensure the quality of the delivery system;

■ For application operation and maintenance personnel , the MeterSphere platform can perform verification after business changes, improve verification efficiency, and speed up the launch of business updates. In the daily inspection of the business system, the MeterSphere platform also plays a very important role. When a system problem is found, MeterSphere can issue a unified alarm in a timely manner, and push the alarm message to the person in charge on WeChat, etc., reducing the duration and scope of the impact of the problem and helping to improve the stability of the entire system.

Community sharing｜Guosen Securities' system quality assurance practice based on open source tools

Guess you like