Database products are emerging in endlessly. How should the financial industry choose?

12.23 Yuan Chuanghui·Shanghai Station, let’s talk about LLM infrastructure

As we all know, the financial industry has extremely strict standards and requirements for data. Especially as online and real-time business scenarios increase, the financial industry is also facing multiple challenges: it must not only meet the high performance and efficiency requirements of real-time data analysis, It is also necessary to ensure data security and integrity. Based on this, the financial industry will be particularly cautious and difficult in selecting data services. Especially now that various database products are emerging in endlessly, how should the financial industry choose? What kind of products are more in line with the future development of the financial industry?

At the FCon Global Financial Conference, InfoQ also interviewed the COO & co-founder of Feilun Technology. He has long been involved in the fields of big data, infrastructure and cloud computing. He has been working at Baidu for the past 12 years. He once served as vice president of Baidu Intelligent Cloud and general manager of big data and cloud storage departments, developing dozens of cloud products and billions in revenue from scratch. Currently committed to promoting open sourceApache Doris, and creating a real-time data warehouse product SelectDB based on the Apache Doris kernel to help China Postal Savings Bank, UnionPay Commerce, and Ping An Life Insurance Many leading financial companies have upgraded their real-time data warehouse platforms. He shared his views during the interview on the pain points and solutions for real-time analysis scenarios faced by customers in the financial field.

Q1: In what areas does the financial industry’s demand for data services mainly focus? Including what are the main demands for real-time data warehouse?

Lian Linjiang: The financial industry is relatively advanced in digital transformation. Both the investment in technical strength and the use of new technologies are very in-depth, but the focus is Digital transformation still faces many demands and challenges.

From the perspective of business needs, we can see that more businesses in the financial industry are moving online. Consumer financial services, corporate financial services, etc. are all APP-based. These online changes have brought about business perception, risk control, and customer insights. and decision-making and other business paths require real-time operations, so more real-time actions such as consumer credit, fraudulent transaction identification, customer behavior insights, etc. are needed. It can be seen that the time and path for the financial industry to serve customers are shorter than before, which requires faster data analysis and response speed.

From a technical perspective, new technologies bring more possibilities for business changes to the financial business. Advanced construction of technology and prediction of cutting-edge technology trends are also particularly important. Therefore, we have seen that many industry customers have begun to set up financial technology Strategic organizations such as the Ministry of Finance and the Ministry of Information Technology will be used to comprehensively promote the financial technology strategy.

However, in many financial industries, there is still lag and complexity in the construction of big data. For example, many financial companies have replicated the big data construction path taken by the original Internet companies to a certain extent, built a big data platform based on Hadoop, and built a large number of systems on top of it to meet the challenges of different businesses. , such as batch processing and analysis systems, real-time processing and analysis systems, etc., ranging from a dozen to dozens. Therefore, from the perspective of technical planning and development trends, the big data system in the financial industry needs to be simplified, and the architecture needs to be simpler and more efficient.

In addition, the technological development of big data is changing with each passing day. How to keep up with the changes of the times is another challenge faced by the financial industry.

Q2: So, how do companies in the financial field find a database product suitable for their own business? Can you give some advice from the perspective of real-time data warehouse selection?

Lian Linjiang: First, start from demand. I just mentioned that business in the financial industry is beginning to move online, and a real-time database is needed to deal with business challenges. At the same time, it also needs to solve a series of problems that come with how to use it well, such as how to integrate data, how to manage it, and how to face it. Further optimization of business, etc.; second, clearly understand future technology trends. The rapid changes in technology often bring about constant updates. This is actually a process of continuous iteration, because the construction of the system lags behind the development of technology. This may cause a situation where efforts are invested in system upgrades today, and in a year or two, they will be upgraded again. Iteration is required under the impact of new technologies. Our judgment on the future trend of big data is to develop in three directions: real-time, unified and cloud-native. Therefore, it is also recommended that enterprise users in the financial field choose products that are future-oriented and in line with technological trends; third, pay attention to products of openness. The so-called openness refers to choosing products that can represent industry standards as much as possible, similar to standard interface languages such as SQL in the database field. Such standards bring more open choices and the inheritance of future historical assets. Looking at big data technology, open source has actually been leading the development of the big data industry. Open source can promote standards very well and can also bring openness.

Q3: Mr. Lian also mentioned the factor of open source. We know that SelectDB is developed based on the open source Apache Doris. For the financial industry, is open source one of the important considerations for enterprise selection?

Lian Linjiang: I see many companies in the financial field today. They generally have a very strong sensitivity and openness to cutting-edge technologies. From the actual communication point of view, everyone's recognition and adoption of open source technology is also generally consistent. why?

First, as just mentioned, open source itself can bring about standardization very well, because open source is a market economic behavior among the developer community. If a good open source product is recognized by everyone, it can lead to wider and wider use. This process naturally has great credibility and standardization, so excellent open source products must have its unique advantages and universality. Adaptability; secondly, if an open source project wants to develop well, it must have continuous advancement, which will also bring continuous and vigorous vitality to the product; thirdly, finance has relatively high requirements for autonomy and controllability. And because open source code can be shared, it has independent and controllable characteristics. If an enterprise has the ability to control it well and invest in construction, it can get the benefits of one plus one in this community.

I think open source is a very promising platform construction path for financial companies, which can provide stronger vitality and positive circulation. Through open source, our products can also be tempered, which is also an opportunity for us. Just like Apache Doris was forged from Baidu's massive data scenarios, it has widely absorbed the needs of multiple industries and scenarios through open source, allowing it to flourish faster. So we can see that companies and developers in the financial field are very supportive of open source, are willing to invest, and are continuing to build.

Q4: As we all know, finance is an industry with extremely high data requirements, so it is also a competitive highland for many database manufacturers. Compared with other financial-level databases, what are the core advantages of SelectDB?

Lian Linjiang: From the first day the company was established, we have been very clear about our positioning - Real-time data warehouse a>, real-time is the first requirement of the product.

To achieve real-time analysis of data, the most important thing is to solve two latency problems, low latency of data integration and low latency of data query. In other words, the data warehouse must be able to process data quickly enough and be visible in real time to support second-level queries.

Therefore, we have made a lot of technical innovations in real-time, including supporting millisecond-level real-time data writing, a primary key storage model with real-time additions, deletions, and modifications, a real-time appended detail and aggregate storage model, and millisecond-level lightweight table structure updates, etc., which can Real-time data import and real-time visibility. In terms of real-time query, SelectDB has extremely fast performance on various query loads such as high concurrency point query, large wide table query, complex multi-table association, etc. In the global analytical database evaluation list ClickBench, SelectDB ranks first in the world in terms of performance due to its excellent performance in a variety of scenarios.

In addition to positioning, we need to further understand the development situation of big data. Currently, enterprises generally use the typical lake-warehouse parallel architecture solution, which includes multiple components for batches, multiple components for interactive analysis, and even more than one lake. A warehouse. Based on this, we have put forward the concept of unification, simplifying the current complex architecture and reducing data components as much as possible; it is particularly worth mentioning that we are also constantly improving the integrated lake and warehouse solution, using SelectDB's modern data platform solution to integrate The data warehouse and data lake are integrated and unified to provide unified data processing and analysis capabilities for various business loads such as BI reports, Adhoc analysis, and batch and incremental ETL in one architecture.

In addition, customers who have cloud needs will pay more attention to the cost performance and resource flexibility of cloud services. SelectDB has also regarded cloud products as its core from the beginning. In October last year, we launched the first cloud-native product, which is also the first cloud-native data warehouse in China based on multi-cloud and fully SaaS-based, SelectDB Cloud. It currently supports major domestic and foreign cloud vendors such as Alibaba Cloud, Huawei Cloud, Tencent Cloud and Amazon Cloud Technology.

In addition to the advantages mentioned above, SelectDB also has the characteristics of simple architecture and rich ecology. When financial customers want to migrate historical assets to SelectDB, the migration and integration of enterprise user data can be well guaranteed. Considering that the big data systems of many financial customers are interconnected at the upper and lower levels, SelectDB has also established product compatibility, mutual authentication and solution integration with dozens of partners.

Finally, due to the particularity of financial customers, continuous companionship and service capabilities are more important. At this point, we have actually made a lot of construction and investment. Currently, we have 7 branches in China, and we will arrange pre-sales, after-sales and other support personnel to provide them with reliable service guarantees.

Q5: Compared with other real-time analysis demand scenarios, do application software in the financial industry have any additional concerns? What solutions will SelectDB use for protection?

Lian Linjiang: For the Internet industry, they prefer to purchase SaaS products one-stop on the cloud, which can be used out of the box. There is also good linkage between products. However, for financial companies, due to reliability or regulatory requirements, a large number of system constructions are privatized and deployed independently. In this regard, we have done a lot of work on financial enterprise-level products:

First of all, we have created an enterprise version for the financial industry, which can be privatized and deployed in various environments, such as virtual machines, physical machines, cloud native infrastructure or private clouds. We can provide them with very efficient deployment, simple and Easy to use, easy to operate and maintain. Secondly, financial customers have very high security requirements for data and the entire IT infrastructure. In addition to ensuring the high reliability, high availability and complete permission system of a single software system, we have especially strengthened our disaster recovery and backup capabilities. It provides CCR capabilities between local dual clusters and multi-location multi-center clusters. Once a service is disconnected, it can be started in seconds and minutes.

Q6: Is it convenient to share an implementation case of SelectDB in financial scenarios?

Lian Linjiang: SelectDB serves many customers throughout the financial industry, including banks, securities, funds, etc. Here I can share a practical case of a major state-owned bank in financial anti-fraud.

Because large state-owned banks have a large number of outlets and customers, doing business online on this basis requires a lot of risk control judgment and processing before, during and after the event. Especially for anti-fraud activities, losses may not be recovered after a day, so basically a feedback loop of seconds or, at worst, minutes is required. In addition, since anti-fraud activities mostly occur at the terminal, major state-owned banks have tens of thousands of outlets and hundreds of millions of users, which require tens of thousands or even hundreds of thousands of concurrent users, which places very strong technical requirements. In addition, as a construction platform, it also needs to be simpler to manage, the data is highly reliable, and every data statistics is accurate. These characteristics determine that its selection is very demanding, so they also conducted a lot of evaluations. In the end, they overall believed that SelectDB's technology was the most in line with the requirements, and its performance was several times or dozens of times higher than that of the same product in the industry.

Now customers have actually implemented it and the results are very good. If the old architecture is used to implement it, the effect may be hourly or even T+1. Now it can achieve second-level real-time performance, so it is also vigorously promoting larger-scale use. They are also planning more implementation scenarios, and also want to use our technology for log analysis to replace the original system for indicator observation and order analysis and query, and the overall cost investment only requires one-third of the previous solution. to one-fifth. Overall for this customer, SelectDB not only satisfies the needs of the business side well, but also better meets the needs of the construction side.

Q7: In the future, how will SelectDB serve more customers in the financial field? Based on this, do we have corresponding plans?

Lian Linjiang: From a technical perspective, we will continue to invest and make progress in the three directions of real-time, unified, and cloud-native, which can well satisfy the needs of the majority of enterprises. Customer needs; today, a large number of financial customers have also benefited from it, and we will continue to make deep technological innovations hand in hand.

From the business scenario, we will make in-depth optimization along the user's business scenario. For example, for portrait behavior analysis, we design functions and optimize business processes; for data analysis, we do real-time reporting, auxiliary decision-making, log analysis, and even AI data analysis. These are more in-depth scenario-based thinking and implementation practices. This means that our technology and business are a two-way iterative process.

In the financial field, there are currently many data analysis technologies and business scenarios. Among them, data storage and data processing are actually very basic requirements. On top of this, meeting the needs of application scenarios requires end-to-end The implementation of solution capabilities requires working together with the vast number of ecological manufacturers in the field. For example, we have jointly conducted indicator analysis with some BI vendors, and the effect has been improved many times compared to before. These scenario solution capabilities will eventually release benefits in a wide range of financial customer scenarios. In the future, we also hope to work with more partners to provide more end-to-end scenario-based solutions.

write at the end

As a commercialization company of Apache Doris, one of the most active open source communities in the global database and big data field, we have seenSelectDB in real time We have made firm investment in the direction of globalization, unification, and cloud-nativeization. We also look forward to the continued deepening of the commercialization of SelectDB in the financial field and the continued addition of end-to-end financial joint solutions, which will help more companies in the financial field unlock the value of data in the future.

Database products are emerging in endlessly. How should the financial industry choose?

write at the end

Guess you like