Business Profile
Kingdee Guanyiyun is a subsidiary of Kingdee Group that focuses on providing e-commerce enterprise management software services. It was established in 2008 and is one of the earliest e-commerce ERP service providers in China. It has established cooperative relations with 300+ mainstream e-commerce platforms , driven by enterprise data, deeply integrates online and offline data, provides more than 110,000 customers with omni-channel management solutions and business-finance integration solutions that realize the integration of business, finance, and taxation, and SaaS covering the entire process of e- commerce ERP one-stop e-commerce management solution, intelligent three-dimensional warehouse management system and e-commerce website system using different business scenarios, etc. , help enterprises to improve data detection efficiency by 180%+.
While having a huge number of users, Guanyiyun ERP system is also faced with massive data analysis and extremely fast exploration needs, so as to help enterprises achieve faster business growth. However, due to the limitations of previous technical solutions, the technical solutions we provided could not effectively respond to business needs, and business stability was also adjusted. As a result, enterprises could not make full use of the value of data to provide better business support, and instead spent IaaS costs. became the main expense. Due to these pain points, we have upgraded the underlying database to the real-time data warehouse technology DataWorks+Hologres+Flink on the basis of the original, which helps enterprises to respond to data queries in seconds, makes business needs more agile, and saves 50% of monthly IaaS fees. %.
Through this article, we will introduce the best practice of Guanyiyun ERP system based on real-time data warehouse, so as to help more enterprises improve the efficiency of data exploration and promote efficient business growth.
Business pain points: low efficiency and high cost
The following is the business structure of Guanyi Cloud ERP system. The main business processes include: e-commerce platform connection (data download and storage), with a huge amount of basic data; order processing process (data review, printing, distribution, delivery), The frequency of data calls is extremely high; report analysis and summary (analysis and summary of commodity, inventory, order, profit and other data), data analysis and computing power will face great challenges; integrated gateway (Qimen, API and other interface data processing) , the data can be processed through precise interface conversion; data interoperability among three-party systems (financial systems such as Kingdee and UFIDA, express delivery systems, etc.), three-party data interoperability is extremely easy to cause problems such as data leakage and capture. For customers in the e-commerce industry, the accuracy of data and the efficiency of processing are very important, especially now that the entire e-commerce industry has entered the era of business finance and taxation, any data error or omission of an order will cause the entire industry to Finance-related departments spend a lot of manpower to review.
As can be seen from the figure above, the services provided by Guanyi Cloud require efficient analysis of data, which is very dependent on data storage and analysis. The original underlying architecture used RDS for MySQL and PolarDB for MySQL. PolarDB is mainly used for Data analysis of storage transactions, such as analysis of orders, members, and commodities; RDS is mainly used to store data that is relatively infrequently read and written, such as: procurement, allocation, adjustment, inventory, etc. However, due to the continuous growth of stock data, this technical solution faces great challenges, mainly including:
- Because the business is growing, the amount of data is also facing rapid growth, but the query efficiency is getting slower and slower, which cannot quickly meet the needs of the business for real-time data retrieval, which affects business decisions.
- After the amount of data increases, machine resources can only be superimposed continuously, resulting in continuous increase in computing costs and storage costs
- Due to the huge amount of data, even a simple database DDL operation takes a long period of time, which has become an urgent challenge for Guanyi and the production and research team.
- Operation and maintenance are troublesome, and business stability also brings great challenges.
Therefore, in order to better support the business needs of Guanyi Cloud, we must consider adopting more advanced technologies to realize real-time writing, updating and real-time query of massive data, and at the same time balance the cost.
New architecture: upgrade to real-time data warehouse Flink+Hologres
In order to solve these challenges, Guanyi's technical team has been actively evaluating and exploring new technical solutions, including Alibaba Cloud XDB, TiDB, Hadoop, etc. However, in the process of exploration, it is not all smooth sailing. On the one hand, the cost of trial and error is too high, and every adjustment may have an impact on business stability and customer experience. On the other hand, the existing relevant cases also lack examples of the characteristics of the e-commerce field, such as massive data processing, high concurrency, and real-time updates. After some difficult explorations, by chance, Guan Yi saw a case sharing in the logistics industry. In the case, real-time data warehouse Flink+Hologres was used to support real-time query and real-time monitoring of logistics orders. Although there are considerable differences between the logistics industry and e-commerce, logistics, as an important part of the e-commerce field, has many similarities in its technical characteristics with the e-commerce field, and we can also make some references. Therefore, Guanyi also began to choose Alibaba Cloud Hologres as a solution. The main reason for this decision is that we have conducted in-depth research on Hologres, and the following features are more in line with our business needs:
1. Distributed system: native distributed system, self-developed storage engine and computing engine, can realize high-performance data writing, updating and high-concurrency query, and the query efficiency is high.
2. Easy to expand: The natural storage and computing separation architecture can dynamically increase storage and computing capabilities according to business needs, meet the high and low peak traffic attributes of e-commerce business, and reduce cost pressure.
3. High security: Data is stored in the Pangu distributed system, which naturally has 3 copies. At the same time, Hologres itself also supports security capabilities such as data desensitization and IP whitelisting, which improves data access performance and availability.
4. High isolation: There is a resource isolation solution for shared storage, which can meet our isolation requirements for different businesses and scenarios, and improve system stability.
In order to ensure the feasibility of the Hologres technical solution, Guanyi has repeatedly engaged multiple business departments to participate in discussions to determine specific application scenarios. The purpose of this is to avoid the impact on user experience due to database switching. After many discussions, it was finally decided to use the report query scenario as the verification scheme to start execution.
The reason for choosing the report query scenario is that Guanyi currently has a huge amount of report data (about hundreds of terabytes), and the query scenario is very complicated. It often takes more than 30 seconds for customers to query hundreds of thousands of data. In terms of use, it is used very frequently, for example: the customer's operation personnel need to check the sales volume of a certain product within a certain time range from time to time to formulate a promotion plan; the customer's financial personnel need to check various orders, product sales, and profits at any time to make purchases Plans, etc., and the data retrieval generated by these query operations is quite large, and the query speed is too slow for the customer experience will be quite poor.
The following figure is the data flow diagram of the report scenario based on Hologres+Flink after we determine the verification scheme:
- Data sources are stored in RDS, including order, inventory, commodity and other data
- We synchronize the RDS data to Hologres through the whole database synchronization of DataWorks data integration, including offline synchronization of full data, real-time synchronization of incremental data, and merging sub-databases and sub-tables into one Hologres table, etc.
- Then data warehouse layering, ODS-DWD-DWS is performed in Hologres, most of which are done through the minute-level scheduling of DataWorks, and then near real-time queries are provided in Hologres. However, some scenarios have very high requirements for real-time data, such as real-time logistics tracking, so we will use Flink to read the Hologres Binlog for data processing, and then write the Hologres solution, so that the data can be written in real time for real-time query, satisfying some businesses real-time query.
- Hologres provides a unified external query application, including real-time logistics tracking, data staff, real-time large screen, operation analysis, etc., so that we don't have to write another copy of the data to MySQL to provide online services and reduce development operations.
In order to integrate the existing report data into Hologres, Guanyi arranged for 4 developers (including DBA and development) to spend a total of 3 months completing the migration of report data from RDS to Hologres. In the early stage of the project, the R&D staff had some blind spots on the table structure and index settings of Hologres, but through the technical support and problem diagnosis provided by the Hologres team many times, they were quickly resolved and the project progressed smoothly. During the migration process, some business departments reported that there were individual data leakage problems, but these problems were remedied in time. For the vast majority of customers, the migration process was completely unaware, which further strengthened our understanding of Hologres Confidence.
After completing the data and task migration, we began to gradually use Hologres to provide business queries until it was fully launched later. After the production, we found that there are several benefits:
- The delay is reduced to the second level: the query through MySQL used to be at least 30s+, but now the query of the new system is basically within 10s, and the overall response speed of the report query has a qualitative leap. With real-time analysis and aggregation capabilities, we have successfully solved the problem of data statistics delay commonly reported by previous customers. These improvements have enabled Guanyi to gain more recognition and trust among customers.
- IaaS costs reduced by 50%: Because we needed to continuously add resources to improve business query performance, our costs increased dramatically. After changing to a new technical solution, the cost has also been reduced by 50%.
- Business is more agile: Now the main products of the new architecture are DataWorks+Flink+Hologres. Hologres provides a unified query application. Our architecture has become more streamlined, and we can respond faster to some temporary needs of the business. At the same time, we can develop And operation and maintenance have also become more agile, reducing many unnecessary operations, and the stability of report query has also improved a lot.
future plan
In view of the effective verification of the new architecture of Flink+Hologres+DataWorks in the Guanyi report scenario, we will continue to explore more business scenarios based on this architecture and build a larger data center platform. Through integration and Manage massive amounts of data, provide more efficient and reliable data services, realize the comprehensive connection of sales channels, business middle-end and financial systems, and create a more intelligent ERP platform, so that data can empower business value and drive business growth and innovation of enterprises.
-
Hologres 5000CU, 20GB storage free trial, go to trial>>
-
了解Hologres： https://www.aliyun.com/product/bigdata/hologram