insert image description here
In the era of full introspection, people's knowledge learning needs are also increasing day by day, and traditional knowledge acquisition methods can no longer fully meet the needs. A large number of knowledge-paying apps have also emerged, providing users with a more convenient and efficient way of learning.

However, the paid-for-knowledge industry is also characterized by rapid changes and high uncertainty, and users' demands for content and services will be constantly adjusted as the times change and trends change. Therefore, the traditional linear and predictive operation model is gradually unable to adapt to the development needs of the industry, and it is urgent to introduce an agile operation model to adapt to changes and flexibly deal with these "uncertainties".

Compared with the traditional operation model, agile operation pays more attention to data-driven, user experience, rapid iteration and flexible decision-making, and can more quickly obtain and respond to user feedback, adjust operation strategy and optimize product functions, thereby improving user satisfaction and profitability. In addition, it can effectively stimulate team innovation and passion, improve team cohesion and execution, and thus bring long-term development advantages. These are the problems that many knowledge payment apps hope to truly achieve breakthroughs. Let's follow the real case of a top knowledge payment app to understand how it can improve human efficiency step by step, promote agile operation, and stimulate team potential under the premise of ensuring content quality and steady iteration of new functions. Concentrate resources to create business growth points.

1. The other side of innovation and iteration: complicated business systems and heavy back-end pressure

In the process of constantly pushing out the old and bringing forth the new, and creating new content sections and functional models, the number of business systems of this knowledge-paying App is also rising, gradually forming a spaghetti architecture that is difficult to maintain, and the consistency and real-time performance of data synchronization are both in place. face the challenge:

① Difficult to synchronize across systems

Faced with a large number of different existing systems, especially some of them are strongly related to each other, or a certain business data stored in different databases, such as financial data stored in different systems, part of it is in MongoDB, and part of it is in In MySQL, it is necessary to rely on heterogeneous data synchronization capabilities to achieve cross-system data synchronization. During the synchronization process, modeling and calculations are also required. In such scenarios, data consistency requirements are the most important. However, the historically used cross-system synchronization solution is database double-writing, which is difficult to provide stable guarantees in terms of data consistency and data quality.

② High pressure on back-end maintenance

As the number of services that the back-end team needs to maintain continues to increase, whether it is downsizing or replacement of personnel, it will bring new problems. The former leads to a surge in per capita pressure and the difficulty is more prominent; Differences in code habits lead to further upgrades in maintenance complexity. Therefore, there is an urgent need to use useful tools to solve this business pain point, relieve labor pressure, and achieve cost reduction and efficiency increase.

③ Negative impact on user experience

For professional apps that serve learners, the pressure of operation and maintenance of the back-end system is directly reflected in the daily experience of users. Taking the recommendation algorithm as an example, the application initially recommends content for users based on regularly updated data, but one of the drawbacks of this operation is that the data changes within the two update intervals cannot be applied to the recommendation algorithm. On the other hand, because the user’s browsing information and behavior records cannot be pushed to the algorithm engine in real time, the best recommendation opportunity is missed, resulting in the loss of business opportunities. In short, due to the separation of the algorithm's engine database and back-end data, data delays have seriously affected the accuracy and timeliness of recommendation results, resulting in a decline in user experience. The same is true for the message system and the review system. The two systems are independent of each other. However, in view of the timeliness requirements of review feedback, the consistency and real-time performance of data synchronization between the two systems are also directly related to user experience. In order to effectively deal with such problems, the team's internal requirements for real-time data are also getting higher and higher. In addition, as the amount of data continues to expand, the time-consuming method of running a full update once a day is also increasing, and it is gradually unsustainable. It is urgent to find an alternative real-time synchronization solution.

In order to quickly open up various systems from the back end, realize real-time and accurate synchronization of data, optimize the operation and management model while effectively saving labor costs, and save energy for subsequent more content and model optimization, the team decided to find a heterogeneous data The real-time synchronization function and the tools of real-time computing power are used to build a new data base solution-this is also the reason for the final cooperation between the App and Tapdata .

2. Tapdata helps build a real-time data platform: allowing data to be stored and flowed on demand

Before formally choosing Tapdata, we also compared and analyzed some common similar tools on the market. Unfortunately, none of them met the needs, especially in terms of the breadth of data source support and the end-to-end full-link real-time capabilities of data. Tapdata’s advantages Especially outstanding, it can meet our needs with a relatively high cost performance. At the same time, during the two years of cooperation, we have also gained a very good experience in terms of timely response to demand and other after-sales services.
——A head knowledge paid app

As a real-time data integration and data service platform built with low-latency data movement as its core advantage, typical use cases of Tapdata include database-to-database replication, data ingestion into data warehouses or data lakes, and general ETL processing, whether from functional modules From the perspective of real-time, easy-to-use and other characteristics, it is highly compatible with the actual needs of the application at the moment.

Tapdata solution: tailor-made real-time data platform

insert image description here

Mirror layer Mongo: used to store data history change records
Kafka: Unified data access for downstream business systems

overall plan

As shown in the figure above, Tapdata's real-time data solution helps data resources to be stored and flowed on demand:

The application team hopes to reduce user operating costs through precise push, and tap user payments. Accurate push pays attention to timeliness. According to internal research and judgment, within 1 hour after the user makes the first transaction, there will be a high probability of secondary payment. This requires the recommendation system to make accurate analysis in a short time and push the matching one. knowledge content. Based on the analysis and dismantling of the above business needs, Tapdata combines its own product capabilities and puts it into an executable technical solution as follows.

First, tapdata is used to synchronize the data of the source business system library in real time, which includes core business data such as order transactions, commodity status, customer behavior records, and message systems. Afterwards, the data falls into the mirror layer, which is used to record the historical changes of all data, and then pushes the mirror layer data to the downstream Kafka in real time through Tapdata. Finally, after the recommendation system consumes Kafka data through Flink, it completes real-time calculation and analysis.

Feedback

With the support of the real-time data platform solution provided by Tapdata, the App has successfully broken through the bottleneck problem. Starting from the source of data, it has truly realized cost reduction and efficiency increase, and facilitated quick and easy upgrade of operation management:

Save labor costs: 2-3 data development, operation and maintenance personnel are streamlined to promote the flow of manpower to business innovation; provide reliable and reusable data results, no need to write new codes for new requirements, and can be directly configured and used, reducing Code maintenance pressure.
Improve data synchronization efficiency and ensure data synchronization quality: Based on Tapdata's powerful real-time data integration capabilities, while improving data synchronization efficiency, it also provides guarantee for data quality and successfully solves the historical problem of data consistency
Optimize user experience and promote user conversion: Personalized recommendations are more real-time. According to browsing records and reading habits, user needs can be accurately captured, real-time recommendations can be realized, and user loss can be effectively reduced.

Why choose Tapdata?

In the technology selection stage, the standards and direction of the application team are very clear: First, in terms of technical capabilities, it is necessary to perfectly solve the data inconsistency problem left by the traditional data double-writing scheme. At the same time, considering the influence of historical development factors, there are a large number of different database types in the app, and many synchronization tools support relatively few data sources. The new solution requires cross-database synchronization and the breadth of data source support. Has a very good performance. Second, in terms of follow-up maintenance, the new tool needs to support batch management data synchronization links, and the operation is simple, which helps to reduce back-end pressure and release human resources.

Faced with the above requirements, Tapdata shows the following advantages:

Out-of-the-box and low-code visual operation

Tapdata is easy to deploy, and supports no-code and low-code visual operations, and can quickly create tasks in drag and drop, without coding or even SQL to write conversion rules.
Built-in 60+ data connectors, stable real-time collection and transmission capabilities

Collect or synchronize the latest data changes from various data sources in a real-time manner, including databases, APIs, queues, Internet of Things and other data providers. Support bi-directional synchronization of multi-source heterogeneous data, and automatically map relational to non-relational. Based on the self-developed CDC log analysis technology, zero-intrusion real-time data collection, almost no impact on the source database, one-click real-time capture, update within milliseconds. It has built-in 60+ connectors and is constantly expanding, covering most of the mainstream databases and types, and supports custom data sources, has a highly scalable PDK architecture, and can quickly connect to the SaaS API system within 4 hours; quickly connect to the database system within 16 hours .
Real-time data computing capability with second-level response

Full-link real-time, based on Pipeline streaming data processing, to meet the real-time processing requirements based on a single data record, such as database CDC, messages, IoT events, etc. Different from traditional ETL, every new piece of data generated and entered into the platform will be responded, calculated, processed and written into the target table within seconds. At the same time, it provides statistical analysis capabilities based on time windows, which is suitable for real-time analysis scenarios.
Stable and easy-to-use data real-time service capability

Supports low-code visualization to develop and configure business-needed Data APIs, and can provide real-time interactive data access capabilities with millisecond-level delays and large concurrency, so as to truly support TP-type services. It has complete and configurable data access rights, supports access monitoring and analysis capabilities, and can provide authority-based self-service master data access services and mechanisms for data demand departments. It has a high-availability and scalable architecture design, which is enough to deal with large concurrency and large traffic access.
Classification of data and tasks allows data to flow across departments.

Supports task classification, and can customize labels according to different projects, which is convenient for quick screening and search, and is helpful for collaborative management and follow-up maintenance of all tasks.
Platform-level data verification

Through a variety of self-developed technologies, the high consistency between the target data and the source data is guaranteed. It supports various verification methods such as bar number verification, primary key verification, row-level verification, and advanced data verification, and completes consistency with different verification cycles such as timing verification, polling verification, and minute-level dynamic verification. Calibration to ensure production requirements. At the same time, it supports the secondary verification of wrong data and the repair of wrong data to jointly provide guarantee for data consistency.
Visual task operation monitoring and alarming

Contains 20+ observability indicators, including full synchronization progress, incremental synchronization delay, etc. It can monitor the latest running status and log information of running tasks in real time, and supports task alarms. Enter the Tapdata console kanban to see the running status of tasks at a glance.

3. Wide table construction and master data management: more possibilities of knowledge payment application x Tapdata

From the above content, it is not difficult to find that at present, the initial attempt of the paid knowledge app and Tapdata at the real-time data platform level has achieved good results and good feedback, but this is only a part of Tapdata's overall product capabilities. In order to further optimize the app experience and design and select higher-quality knowledge content for learners, Tapdata is also continuously looking for more connection points between knowledge apps and other functions of Tapdata, such as the planned wide table construction and master data management.

Wide table construction: Improving data processing efficiency and data utilization value

Knowledge payment apps, especially such top products, usually need to process a large amount of user behavior data and content data. In order to better analyze and utilize these data, it is necessary to consider the association between data tables and how to maximize the efficiency of data reading when designing the data model. As a data table design method, Tapdata’s wide table construction capability supports merging data in multiple associated data tables into one large table, reducing associated queries between tables, thereby effectively improving query efficiency, and can be used for users 360, Course resource management, order management and other scenarios.

The wide table can help operators better analyze the relationship between user behavior and content data, such as courses purchased by users, videos watched, content of messages, etc. Through the analysis of these data, knowledge payment applications can better understand The needs and interests of users, so as to provide better content and services, improve user satisfaction and retention rate. At the same time, wide tables can also be used to generate data reports, recommend algorithms, etc., to improve the utilization value of data.

Master data management: improve data quality and service quality, optimize user experience

As a data management method, master data management (MDM) supports the centralized management of key information in data to ensure data consistency, accuracy and reliability. For knowledge-paying apps, effective master data management means a comprehensive upgrade of data value, data quality, service quality, and data security, which helps to further improve user experience:

Data consistency: Knowledge payment apps usually have multiple data sources, such as user information, course information, payment records, etc. These data sources may be duplicated or inconsistent. Master data management can ensure data consistency and avoid An error or confusion occurs.
Data integration: Master data management can integrate data from different data sources into a single data source, which facilitates data analysis and mining for knowledge payment apps and improves data value.
Data quality: Master data management can standardize and clean data to improve the quality and accuracy of data, thereby improving the service quality and user satisfaction of knowledge payment apps.
Data security: master data management can control the authority and security management of data, protect the core data of knowledge-paid apps from illegal access and theft, and avoid data leakage and loss.

As a real-time data platform with its own ETL, Tapdata has two core technical capabilities of real-time data integration (ETL) and real-time data service (DaaS), which can help enterprises quickly connect to island systems without code and build the base of enterprise master data. With the help of Tapdata master data management, it can help knowledge payment applications manage and maintain data, improve the data quality and decision-making capabilities of Apps, and better meet user needs. It is applicable to multiple scenarios such as knowledge classification management, user information management, payment information management, and content management.

We look forward to more in-depth cooperation between Tapdata and paid knowledge applications in the future, creating more surprises for learners and stimulating knowledge.

【Recommended reading】

Original link: https://tapdata.net/tapdata-enabling-agile-operations.html

Tapdata Empowers Agile Transformation: How Head Knowledge Paid Applications Use Real-time Data to Rapidly Improve Human Efficiency and Create Business Growth Points