Original Author: Zhao Yuying, Editor-in-Chief of InfoQ
At present, a well-known domestic sports brand operates 12 shoes and apparel sports brands around the world, and has nearly 10,000 offline stores across the country. Most of the stores of Nike, Adidas, Puma, Converse and other brands are operated by its agents, with registered members reaching These businesses are fully supported by its technology company Yunsheng Haihong. In the past ten years, Yunhai retail system is a retail service platform supporting omni-channel and full-category sports shoes and clothing, supporting the retail of 8000+ offline stores.
How did such a veteran company in the retail field switch from MySQL to a native distributed database step by step? What is the overall structure change idea? After practice, how to evaluate Oracle and domestic distributed databases from the perspective of cost... Recently, InfoQ had the honor to interview Hong Liang, Chief Architect of Yunsheng Haihong, and discussed the above issues one by one.
background introduction
Before introducing Yunsheng Haihong's database architecture design, let's understand its overall business background. The core business of Yunsheng Haihong is the retail system, including three modules: inventory, terminal retail and financial support system used within the group.
Since 2013, Yunsheng Haihong has started to build the entire database architecture, and has gone through multiple rounds of iterations due to the continuous development of the business. Before 2016, Yunsheng Haihong was basically still in the traditional retail era. The internal information systems were built by the major regions, maintaining their own database structure, and uploading business data to the headquarters every day. The database adopts a centralized single database. The advantage of this method is that the structure It is simple, but its shortcomings become more and more obvious with the development of business. For example, there is no way to check the regional summary data in time, and it is also impossible to check the real-time inventory of the whole country across regions.
In order to solve these problems, Yunsheng Haihong launched a new architecture in 2016 - Yunhai Retail System, which opened the way for the evolution of the architecture in the era of digital retail.
From MySQL to Oracle to full TiDB
Up to now, Yunhai retail system has mainly experienced three stages of evolution.
Phase 1: Apply micro-services, realize data sharing, initially refine operations, and support digital business development
At this stage, Yunsheng Haihong uses the method of microservice + MySQL sub-database and sub-table. At the beginning of the project, the team considered that the data vertical segmentation mode is relatively stable in a short period of time, and the difficulty of MySQL cluster development is relatively easy for the team to grasp, so MySQL was selected.
With the rapid development of the business, many problems exceeded the team's original expectations. The MySQL cluster did not support complex report analysis. The team tried to introduce Oracle to share this part of the demand, and then synchronized the data in real time through Otter to ensure the integrity of the data on both sides. For the TOB business, the internal report is very critical, and the data accuracy is extremely high, and the hot and cold data changes frequently. The introduction of Oracle has solved the problem of real-time reporting.
Since then, Yunhai retail system has supported the rapid development of business for five years, and achieved many small goals, such as realizing the storage of massive data in various regions and major regions of the country, realizing real-time data sharing, and achieving the goal of business visualization. However, with the expansion of business and the increasing difficulty of requirements, some new challenges gradually emerged. First of all, the entire architecture is based on MyCAT for sub-database and sub-table. In daily maintenance, if there is a new business, such as adding a table or adjusting a table, the maintenance level will increase labor costs, and the configuration needs to be adjusted manually, and then the configuration will be called. A lot of energy.
Secondly, there were already 110+ Otter synchronization channels at that time, and it was not so ideal to use. For example, adding a table at the source end, but not adding a table at the target end, or simply adjusting fields may also cause some synchronization interruptions, which require a lot of manpower to maintain. The most important thing is that Oracle has also encountered some bottlenecks, such as the inability to expand massive data and the poor analysis time of the aggregation library.
Phase 2: Solve the problem of poor timeliness of aggregation library analysis due to explosive data growth
Before 2020, Oracle's single-point performance could no longer scale horizontally, and the team began to actively seek alternatives. At this time, the team started to get in touch with TiDB, and heard a detailed explanation from Huang Dongxu, the co-founder and CTO of PingCAP at the ArchSummit conference held by InfoQ at that time. In terms of query performance of complex SQL, I found that TiDB can solve the problems of Oracle and is very convenient. After the internal small-scale trial achieved remarkable results, Yunsheng Haihong finally decided to quickly promote the deployment of the TiDB cluster.
Decide on the stress test plan when deploying TiDB to production (using Percona's open source tool Percona-playback to implement the stress test)
"In 2020, the epidemic broke out, which had a great impact on our business. We began to focus on online business. The most direct pressure on the technical side came from the change of the inventory management module. Originally, from receiving the need to connect with Taobao , Jingdong, Vipshop, Douyin and other platforms will take three months or even half a year to finally implement, but because we have already switched to TiDB in the early stage, we have made sufficient preparations at the technology stack level, and it only took two weeks in the end The adjustment of the single-platform inventory management module was completed within a short time," said Hong Liang.
Architecture diagram after the introduction of TiDB in 2020
As far as internal engineers are concerned, the deployment of TiDB has also progressed very smoothly. First of all, Yunsheng Haihong's main business is built on the basis of MySQL. TiDB is fully compatible with the MySQL protocol, and the migration from MySQL to TiDB is relatively smooth. Secondly, TiDB’s daily operation and maintenance, capacity expansion, and capacity reduction are very convenient. In the past, the DBA needs to complete the data migration of more than a dozen instances in the early morning on a monthly or quarterly basis. The maintenance workload is huge, and the risk of data migration is extremely high. The consequences of problems are very serious. After the introduction of TiDB, there is basically no need to perform migration actions, not to mention the time-consuming actions of MySQL daily inspection, archiving, and backup. Finally, the limitations brought by MySQL sub-database and sub-table cannot allow the team to quickly respond to changes. Every adjustment of the company's organizational structure will have a certain impact on the business. The team needs to quickly absorb this impact. The introduction of TiDB makes the entire technology stack more efficient. flexible.
Phase 3: Moving towards the full deployment of distributed databases, and initially exploring cloud-based architectures
At present, Yunsheng Haihong has completed the migration from MySQL to TiDB internally. From the initial version 4.0 to the current online version 5.4.2, every upgrade of TiDB will bring more practical features and functions. Next, Yunsheng Haihong will try to migrate from Oracle to TiDB, gradually gather database clusters, and further reduce the burden of operation and maintenance. Within Yunsheng Haihong, Oracle will not undertake too many core businesses and write operations, and migrate basically AP-oriented data and businesses, so this part is relatively easy. The team will focus on front-end data migration, including data accuracy calibration. test.
In the interview, Hong Liang said that the current internal TiDB cluster has reached 100 machines, and two TiDB clusters have been deployed to bear the business load of the front-end and back-end respectively. The AP business mentioned above is the current financial statement analysis load undertaken by Oracle. At that time, all the business of Yunsheng Haihong will run on the TiDB cluster, and the Oracle cluster will be gradually shut down.
In addition, the overall architecture will gradually become cloud-based. At present, some applications of Yunshenghaihong have been transformed into private clouds, and in the future, some environments will be tried to be transformed into public clouds, such as development, testing, training, production, etc.
Discussion on the Core Problems of Database Design
In the retail industry, Yunshenghaihong can be regarded as one of the companies that invest heavily in technology, and combined with its business scope and volume, it is difficult to build a technical architecture. There are many factors to consider in database selection and architecture evolution. In the process, the team also explored some experience.
Is it possible for the retail industry to completely abandon Oracle?
In the retail field, companies with a certain history must have deployed Oracle databases in the early days, especially for financial data that required extremely high precision. At that time, there were not many domestic databases that could be replaced. Nowadays, domestic databases are becoming more and more mature, and there are more and more options to choose from. Many companies have begun to try to migrate to other databases.
Judging from Yunsheng Haihong's experience, the retail sector has every opportunity to abandon Oracle in the future, and even the processing of extremely demanding financial statement data can be handled by domestic databases.
In terms of model selection, enterprises need to do pressure testing in advance according to business characteristics, and also need to make relevant plans before migration. Yunshenghaihong has made sufficient records from MySQL to TiDB, and from Oracle to TiDB.
From a cost perspective, are distributed databases worth it?
Now when it comes to cost, it basically covers many dimensions such as software licensing fees, software service fees, hardware procurement fees, and daily maintenance fees. There are also differences in different internal situations of enterprises.
Judging from Yunsheng Haihong’s experience, TiDB definitely has a clear advantage over Oracle in terms of software licensing fees; in terms of software service fees, TiDB’s own ecology and community construction (including documentation) are relatively complete, but some domestic Due to the lack of maturity of the database, manpower cannot be invested in building a mature service ecosystem. This needs to be judged according to the selection situation; in terms of hardware procurement costs, Yunsheng Haihong has little difference before and after use; in terms of daily maintenance, TiDB has a low threshold, Easy maintenance saves a lot of labor costs.
Compared with managing MySQL clusters, data backup, hardware fault handling, master-slave node management, etc. are relatively troublesome, but TiDB can basically achieve lightweight maintenance, and the later cloudification may further reduce operation and maintenance costs.
Do you want to fully cloudify?
As mentioned above, Yunsheng Haihong will actually gradually cloudify in the future, and its team has a lot of considerations about this.
In the interview, Hong Liang said that from the perspective of the entire cluster rather than a single database, cloudification will have more advantages than local deployment in terms of computer room management, network security, high availability, and disaster recovery. Today, TiDB and Alibaba Cloud also cooperate, and cloudification is relatively easy, especially for enterprises whose original technology stack is based on MySQL.
Is the value of intelligent operation and maintenance worth considering in the early stage?
In the past two years, many databases have been actively integrating AI capabilities in order to make the whole process of deployment, operation, operation and maintenance more intelligent. For Yunsheng Haihong, the demand for implementing AI within the enterprise is relatively less urgent.
"Intelligent operation and maintenance or the introduction of AI capabilities depends on whether the underlying infrastructure is in place. If the storage and calculation are separated or the operation and maintenance capabilities are not improved, AI is like a castle in the air. Only when the underlying foundation is laid, intelligent operation and maintenance can be developed. For example, some indicator monitoring of MySQL is definitely not as perfect as TiDB, without these indicators, AI monitoring would be out of the question.”