With the continuous advancement of digital transformation in various industries and the deepening of information construction, the scale and complexity of IT systems are increasing day by day. According to IDC's forecast, the scale of IT expenditures in China's financial industry (including: software, hardware, IT services, etc.) will reach 218.602 billion yuan in 2021, and will reach 335.936 billion yuan by 2025. As IT resources become more and more abundant, the amount of monitoring data also shows an exponential growth trend, resulting in increasing management complexity and increasing demand for intelligent operation and maintenance.
In the Internet age, various behavioral information, status information, indicator information, etc. will be recorded and stored in the form of logs, indicators, events, etc. Usually, these machine data are scattered in various business systems, and their value Digging is more difficult. As far as the financial industry is concerned, in all business scenarios, data, as an important carrier of various business activities, is electronically stored in massive operation and maintenance data, and its value is immeasurable. These data include not only the performance characteristics of resource consumption at the system level, but also the detailed transaction logs related to customers and accounts at the business level.
As a key means to support the stable and continuous operation of financial institutions' digital business, intelligent operation and maintenance has broad prospects for future development. On the one hand, with the continuous deepening of financial technology applications, the scale of enterprise data has increased sharply, and the bottleneck of traditional process management has become increasingly prominent. With the ever-increasing volume of data, traditional operation and maintenance methods have been difficult to meet the increasingly complex business monitoring and security management needs of financial institutions; on the other hand, through intelligent operation and maintenance methods, financial institutions can effectively monitor operations in different operation and maintenance scenarios. Maintenance resources, and through the application of technologies such as Big Data Analytics and artificial intelligence (AI), intelligent management of the entire life cycle of operation and maintenance can be realized in a timely manner, so as to ensure system stability and business continuity. This is also a financial institution that can basis for sustainable development.
The application of intelligent operation and maintenance in the financial field has become a rigid demand. Banks, insurance, securities and other financial institutions are accelerating the construction of intelligent operation and maintenance systems. Based on data, using algorithms, and adopting observability technology to assist operation and maintenance has become the consensus of current finance and various industries. Combining new technologies of automated operation and intelligent operation and maintenance with data management must take into account some common problems faced by machine data analysis in the financial industry.
O&M Difficulties and Challenges Facing the Financial Industry
► Huge amount of data—difficult to manage
The IT environment includes network devices, security devices, servers, virtual machines, middleware, services, business systems, etc. Data is generated all the time from the underlying hardware to the upper-level software, and the data of various logs, alarms, and indicators can be collected every day. Up to dozens of terabytes, with the passage of time, it is bound to precipitate massive amounts of data. These data contain a lot of key information, such as business transaction logs, server indicator information, system events, abnormal alarms, etc. These data are the basis for finding problems and locating faults. Finding the root cause of failures in massive amounts of data is a long-term problem faced by IT operation and maintenance.
► Data scattered—difficult to unify
Although most financial institutions currently have network monitoring platforms, log management platforms, Zabbix, Prometheus and other related operation and maintenance management tools, a certain tool often only focuses on a certain type of data or a certain scenario. Analyzing and locating problems in the IT environment often requires the cooperation of multiple tools, which causes inconvenient use. More importantly, it causes scattered storage of various types of operation and maintenance data such as log data, event data, indicator data, and alarm data, forming a data island and making it impossible to Data association analysis and unified display.
► Multiple types of data—difficult to analyze
Operation and maintenance data such as logs, events, and indicators often appear in text (string) format, without a fixed format, and vary with different vendors, and most of them are unstructured data. These data cannot be directly analyzed or even read, which is undoubtedly a very big challenge for operation and maintenance personnel. With the rise of artificial intelligence technology, it has become a trend to apply AI to operation and maintenance, use algorithms to analyze, and make decisions instead of humans. It can help companies quickly gain insights into faults and problems that are difficult for humans to reach, accurately predict risks, and automate passive operations. Dimension is active operation and maintenance. However, most of the current operation and maintenance platforms focus on monitoring and only provide query, display, and alarm functions. They have weak analysis capabilities, let alone intelligent algorithm capabilities. Its essence still relies on manual observation, analysis, and positioning of problems, requiring operation and maintenance personnel to have rich operation and maintenance experience and high technical capabilities.
► Troubleshooting—difficult to locate
The IT system is huge and complex. In order to complete a certain task, multiple systems or services need to call each other. When a fault occurs, many systems or services may generate alarms at the same time. The traditional monitoring operation and maintenance platform displays the monitoring data in the form of charts, which can only reflect the abnormality of a certain type or a certain type of data. It presents the problem from a single data perspective. It is like a blind man who can only see a part of it, and it is difficult to observe it comprehensively. The health status of the entire business system. This makes it increasingly difficult to locate faults in multi-layered system architectures. Finding the root cause of a fault that exists in a multi-system architecture in a large-scale fault often requires the cooperation of multiple departments and multiple operation and maintenance experts to investigate one by one. It was time-consuming and labor-intensive, which seriously affected the business experience.
Introduction to Miaoyun Financial Industry Solutions
In order to solve the above-mentioned problems faced by the financial industry in the operation and maintenance process, Miaoyun starts from three aspects and levels:
First of all, build a unified big data operation and maintenance platform , collect a full amount of operation and maintenance data such as logs, indicators, alarms, and events, solve the problems of data dispersion and data islands, and provide a basis for subsequent data analysis and data display.
Secondly, based on the data platform, integrate supervised, unsupervised, and semi-supervised machine learning algorithms. According to common problems in operation and maintenance, the algorithm is applied to daily operation and maintenance scenarios, such as dynamic threshold alarm, alarm convergence, indicator analysis, alarm correlation analysis, intelligent abnormal log detection, root cause location, etc., to build out-of-the-box operation and maintenance Scene algorithm analysis platform.
Finally, the analysis results need to be presented to solve the problem of difficult observation. The solution revolves around the topological diagram of the business system, showing the topological relationship of each component of the system, and displaying data related to the application system such as logs, events, indicators, and alarms, and doing it from a business perspective. O&M makes the business running status observable. Observe, discover, explore, and locate faults on the panoramic business operation and maintenance view.
Miaoyun helps Haifutong Fund to solve the problem of operation and maintenance
Haifutong Fund Management Co., Ltd. was established in April 2003. It is the first batch of Sino-foreign joint venture fund management companies approved to be established in China. As of September 30, 2021, Haifutong managed a total of 89 public offering funds, and the assets of the public offering funds managed by Haifutong were about 138 billion yuan.
Similar to most financial industries, the operation and maintenance of Haifutong Fund also faces massive and scattered operation and maintenance data, which makes it difficult to analyze and locate faults. In daily operation and maintenance, such scenarios often appear: when a business fails, it is necessary to log in to the network device to troubleshoot network device problems; log in to the security device to rule out external attacks; log in to the server to troubleshoot operating system problems; System log, check whether there are abnormalities such as error, warning, and exception. Such one-by-one investigations have very high technical requirements for O&M personnel and the cooperation of multiple people. The positioning time is long and the investment cost is high, which seriously affects the business experience.
Based on the operation and maintenance status and development needs of Haifutong Fund, Miaoyun provides solutions for it that can be roughly classified into three parts
First of all, build a unified operation and maintenance data platform, collect operation and maintenance data such as routers, firewalls, VPN servers, Linux servers, Windows servers, business systems, indicators, events, etc., and provide users with an operation and maintenance data query platform to solve data dispersion and positioning The problem of logging in to multiple systems when the problem occurs.
Secondly, the analysis effect of VPN logs, Windows events, firewall events, and host indicators is displayed in a visual way. Including: VPN destination address TOP10 statistics, VPN source address TOP10 statistics, Windows login analysis, security event classification and proportion statistics, attacker rankings, abnormal event trends, etc., graphically display the information contained in the data, allowing users to view the data Switch to looking at graphics.
Finally, build a business topology map from the perspective of "Haifutong APP", "Fund Supermarket", "Special Account Wealth Management" and other businesses, present components, data and analysis results related to the business on the topology map, and mark faulty components. Visually display business failure points.
Highlights of Miaoyun Financial Industry Solutions
Through the introduction of automation and other management technologies, the unified query and management of the operation and maintenance data of various business objects is realized, and the problem of data islands is solved; through machine learning algorithms, various scattered data are analyzed and displayed in a unified way, allowing users to Observing, discovering and locating problems on one platform greatly reduces the technical difficulty and investment cost of operation and maintenance, and improves the efficiency of operation and maintenance.