Abstract: At the Alibaba Cloud Database Technology Summit, Wang Zhentao, the architect of Inke Live, a special guest, shared the changes in the database structure of Inke Live as a startup company from 0 to 10 million daily active users, the classic application scenarios of databases in live broadcasts, the database storage Optimization ideas, and how to build a highly available database architecture.
On August 24th, the Alibaba Cloud Database Technology Summit came. This technology summit invited Alibaba Group and Alibaba Cloud database veterans to share the first-line database practical experience and technical dry goods. At this summit, Wang Zhentao, the architect of Inke Live, a special guest, shared the changes in the database structure of Inke Live as a startup company from 0 to 10 million daily active users, the classic application scenarios of databases in live broadcast, the optimization ideas of database storage, and How to build a highly available database schema.
The following content is based on the live video and PPT of the speakers.
The content of this sharing will mainly focus on the following four parts:
1. Development History of Inke Live
2. Live broadcast meets cloud database
3. Changes in database architecture on the tuyere
4. Analysis of typical application scenarios of live broadcast
1. Development History of Inke Live
Inke Live is a company established in May 2015. In the field of mobile live broadcast, Inke is a relatively early company. As shown in the above picture are some pages on the Yingke APP, the left picture shows the popular content in the Yingke APP, here is a certain anti-string actor anchor who is live broadcast, and more than 50,000 viewers are watching at this time; The picture on the right is the discovery channel of Inke Live. The discovery channel mainly displays and pushes high-quality content on the platform during popular times. In the Inke APP, there will be many pages similar to those mentioned above, including game live broadcasts, fitness live broadcasts, and small videos. As a company that has just been established for two years, the development speed of Inke Live is still very rapid. From the beginning of Inke's establishment in May 2015, the DAU when the APP was just released was only 200. By October of that year, the DAU was only 200. It has reached 100,000+, and in just two months in December, the DAU of Inke's live broadcast has exceeded 1 million, and in the middle and late 2016, the peak DAU has reached the level of 10 million. The above is the general development process of Inke Live.
2. Live broadcast meets cloud database
Business start: Inke APP released
Next, I will introduce the evolution of the structure of Inke Live from its inception to the present. When the Inke APP was first released, there were actually only three people in charge of the entire backend, two for development and one for operation and maintenance. The architecture at this time is shown in the figure below. The initial architecture is actually very simple. The background is built with 8 virtual machines, 6 of which are used for services, and the remaining 2 are used for storage. The storage part is built by itself. MySQL and Redis. In this initial stage, the most demanding point for the back-end system is that when a product idea appears, it needs to be realized as soon as possible. Such an architecture is very suitable for realizing a live APP under the requirements of the initial stage of the business. At that time, the development work of the first version was only It took 2 weeks.
Business Development: Scale
With the increase in the volume of Inke's live broadcast business, from May to October 2015, Inke's DAU reached 100,000+, and by December it had reached 1 million+. The business development at this time has already After reaching the stage of scale, the overall server-side architecture design at this stage is shown in the figure below. The architecture at this time is also divided into access layer, business layer, base layer and data layer. However, compared with the first version of the architecture, the overall framework has been basically formed, and the architecture of this version has been realized. For some business splitting, data splitting and modular reuse, the bottom layer of the service still relies on Redis and MySQL databases. The most significant change at this stage is the introduction of the cloud, which means that the services at this time have already been migrated to the cloud.
For Inke's services to go to the cloud, the considerations at that time were mainly based on the following points:
1. DDoS, because the cloud host itself can prevent DDoS attacks, so that all the proxy layers can be deployed to the cloud.
2. A natural attribute of live broadcasting is that there are a lot of celebrity events and other big V events, because the instantaneous pressure caused by these events is very large, if you use a self-built computer room, you need to pay for these events that may only last a few hours. A lot of servers are hoarded due to activity, and a lot of reserved resources need to be purchased. After the cloud is used, the cloud host itself is charged according to the usage, so that many core services can be deployed on the cloud, and the capacity can be directly expanded when there is an instantaneous activity that needs to be expanded, and when the activity is over, it can be expanded. By releasing these redundant resources, such elastic scaling can greatly reduce the operating cost of the live broadcast platform.
3. These cloud resources, such as cloud hosts and cloud databases, are deployed in cluster mode, and the cluster mode is often transparent to users, so it can be understood that the cloud itself helps us achieve a layer of high availability.
4. Smooth and scalable. For example, if the current database configuration or performance cannot meet the requirements, you can use these resources on the cloud for smooth expansion, and the expansion on the cloud has no impact on the service, and includes SLB load balancing These devices are friendly to cloud users.
Taking the above four points into consideration, Inke Live uses cloud services at the stage of business scale.
One of the first problems that need to be faced after using cloud services is the overall migration. For the overall migration, the cloud service itself also provides some tools to synchronize the storage in real time and switch at the end. For the cloud service itself, it can be isolated and the image can be packaged and released to the cloud. The business volume of Inke Live at that time was not too large, so overall the process of migrating to the cloud was relatively smooth. But everything may not only have a good side. When using cloud services, there will also be some problems at the same time. Inke also encountered some uncontrollable factors in the process of migrating to the cloud, that is, "acclimatization". For example, the number of MySQL connections for cloud services at that time was limited to 2,000, and many service frameworks used by Inke at that time were multi-process frameworks, which was very easy to cause insufficient connections. The network bandwidth limit is 200M, which is easy to run full, and the same is true for the intranet bandwidth; in addition, some resource preemption or downtime will occur when using cloud hosts. To sum up, there is a good side after going to the cloud, but there is also a side that is not particularly comfortable. In this process, Inke is no longer just a user of cloud services, but has also evolved into a product manager or QA of cloud services. Inke and Alibaba Cloud have also been in contact with Alibaba Cloud from the beginning and learned about "love and kill" later. And work together to promote the optimization of cloud services until both parties reach a better state.
Business explosion: tens of millions of daily active users
In fact, the cloud service itself is used as an infrastructure, which can provide us with high-quality operation and maintenance. However, when the daily activity and traffic of the business are still growing explosively, it is not enough to rely on the infrastructure of cloud services, so the service framework of Inke is also constantly iterating. The architectural framework shown in the following figure is the framework after a relatively large transformation after Inke went to the cloud. This framework has achieved some layers, mainly divided into user layer, application layer and data layer, and in the On the cloud service infrastructure, a lot of technical middleware such as service-based reconstruction and storage optimization have been realized, and various aspects of work have been done to ensure that when the business outbreak reaches tens of millions of daily activities, the service can still be supported. The stability of the frame itself is also quite good. Recently, one of the things that Inke is doing is how to ensure the monopoly and security of resources when the volume is already very large. Currently, VPC is being deployed based on cloud service facilities, and this work is currently progressing. Not bad either.
cloud architecture
In fact, you can see that Inke's overall architecture is built on the cloud service infrastructure. At present, Inke has reached tens of millions of daily active users and is still using cloud services. In fact, it is mainly based on the following considerations: First, we believe that the development of cloud services is relatively mature at this stage, and it is also relatively safe. Yes, cloud services can guarantee some high-availability support internally; in addition, all services used by users are basically clustered and support smooth scaling, which is very important for live broadcast platforms with event operation requirements. Valuable; and users of cloud services do not need to reserve too many resources to support scalable capacity; in addition, if self-built computer rooms are used, the operation and maintenance team needs to know enough about each link, and needs to be able to support from From hardware facilities to monitoring and going online, this is very difficult for the operation and maintenance team of a company with rapid iteration, and the use of cloud services can ensure the high-quality operation and maintenance of the infrastructure. The above are the important reasons why Inke still chooses to build the future on the cloud with the current volume of tens of millions of daily active users.
In fact, many services and systems are ultimately data, so the top priority or foundation of cloud services is cloud database, which is also the technical level that Inke's architecture has been constantly iterating and optimizing. As the foundation, cloud database has also experienced the evolution from the stand-alone era to the cluster era, and then to the distributed framework.
3. Changes in database architecture on the tuyere
Next, I will share with you the history of database architecture changes in the process of Inke from 0 to 10 million daily activities. When it comes to databases, whether it is a relational database like Oracle and MySQL, or a NoSQL database like MongoDB, the first thing that comes to mind is the performance of the database. The performance of the database host is mainly related to the following aspects: CPU, storage, index, IO, lock, memory, and internal threading model, etc. These are all critical aspects to ensure the performance of the database design itself.
At another level, when the database is clustered or distributed, it may face another problem, which is the so-called CAP theory. The CAP theory is that when you want to balance the three aspects of consistency, availability, and partition tolerance, you need to combine specific business scenarios, and you need to selectively give up some things for different business scenarios to protect the core demands. .
In the process of the evolution of Inke's database design, it has roughly experienced the following changes: the number of users has increased from 1,000 to 100 million, the number of daily active users has increased from 100 to 10 million, the number of big V fans has increased from 100 to 10 million, and the peak value of gifts The QPS has increased from 10 to 10,000, and the relational data has increased from 1,000 to 100 billion. You may wonder why relational data has suddenly increased from one thousand to one hundred billion. In fact, the live broadcast itself has some social attributes. For example, when a user likes an anchor, he may follow him, and each follower may generate two records. Therefore, when the number of users has reached 100 million, the amount of relational data will be Very big and very important. Therefore, in such a process, the challenge of storing hundreds of billions of relational data is still very great. In addition, the live broadcast itself also has a big V attribute. Many big anchors may have tens of millions of fans, so every time a big V broadcasts a live broadcast, millions of fans need to receive the big V's start broadcast reminder. For tens of millions of fans, the attention information that this big V is broadcasting live must appear in their watch list. Similarly, when the big V closes the live broadcast, the live broadcast information in the watch list needs to disappear immediately. These requirements for real-time, high concurrency, and large capacity are very high. In addition, a very important point is that the live broadcast of Yingke itself supports gift giving, and Yingke also uniquely supports the continuous delivery of gifts, that is, for example, if I like a host, the value of the gift may be a dime. , but continuous gifting can be carried out, which can be sent tens of thousands of times in a row. Although this is very cool for the user's experience, it will cause a lot of pressure on the server. Therefore, the financial data of the gift-giving of 10,000-level QPS Pressure is also a big challenge that Inke has faced before.
In this process, the collision between basic data and cloud database is involved. On the one hand, basic data is growing explosively; on the other hand, cloud databases need to be able to support the exponential growth of data.
During the evolution of the database architecture, whether it is MySQL, Redis or MongoDB, the database-level services have undergone changes as shown in the figure below, from the single-host mode to the later independent host, to the later master-slave read-write separation, and then according to The dimensions such as user, room, and data are split vertically, and when the dimensions are already large, for example, when the relational data has reached the order of hundreds of billions, it is also split horizontally.
Key technical points - data migration tools
There are some technical points involved in the evolution of database architecture, which can be shared with you here. This part of the data migration tool has always played an extremely important role in the evolution of the database schema. For the migration of homogeneous data, there are actually relatively mature solutions. The migration of homogeneous data is equivalent to the part from the single host to the vertical split in the above figure. This part is due to the structure of the data table and the data structure. There is no change in the splitting strategy, so this part of the work directly relies on the cloud database migration tool to achieve smooth data migration, and the business is basically imperceptible. The applicable scenarios for this part are server-level, instance-level expansion or read-write separation, and vertical business splitting. But when the volume is large enough to require many horizontal splits, many existing mature tools are no longer applicable. During this process, the technical team of Inke Live developed its own set of heterogeneous data migration middleware based on the log level such as binlog. This middleware is more suitable for the migration of heterogeneous databases or sub-database and sub-table. Because in the process of sub-database and sub-table, it is still difficult to be as non-perceptive as possible, especially when it is a single database and single table at the beginning, it needs to be divided into 1000 tables. After 1000 tables are divided, it is very difficult. It is difficult to go back to the past, and the cost is very large. If the heterogeneous data migration components do well enough in this process, the risk can be reduced to a very low level.
For the migration of heterogeneous data, we can share with you a relatively mature solution. For example, we need to migrate from the data dimension of a single database and single table to 32 databases, each with 32 sub-tables, such as the data dimension of more than 1,000 tables. At this time, the data dimension of more than 1,000 tables in sub-database and sub-table can be regarded as the logical repository of the previous single database and single table, and this logical repository is triggered by the basic component binlog at the beginning to ensure the real-time synchronization of data. The delay can basically reach the second level. If so, you can migrate some read operations in the service first, because for read operations, there is no cost to roll back even if there is a problem. After the read operation migration is completed and there is no problem through the verification and comparison, the relatively few write operations are migrated, which can reduce a lot of risks as a whole.
Key technical points - sub-database sub-table middleware
The second key technical point involves the problem of accessing sub-database and sub-table data. In the realization of this part, we also investigated many industry solutions. At present, there are mainly two kinds of mature solutions in the industry, namely Client-based and Proxy-based. The Inke technical team has also implemented these two solutions in different periods, but the general purpose is to achieve sub-database and sub-table. Transparent. First of all, for the business, you don't need to care which database table SQL is executed on, and you don't need to care about the strategy of sub-database and sub-table. The second is the unified management of the configuration of the database itself, which is very useful for the unified switching and configuration of online data. The third is the need to provide unified monitoring. Fourth, since this sub-database sub-table middleware is used, it means that all businesses depend on this middleware, and its high-availability design is very critical. In addition, the design of the sub-database sub-table middleware itself is a cluster mode, so it is necessary to do a good job in the management of the intermediate links between the middleware and the database, and better manage the links through some connection pool models. Including the following load balancing and better scalability, all need to be considered when implementing sub-database and sub-table middleware.
Combining the above technical key points, Inke's technical team has implemented two kinds of sub-database and sub-table middleware. The lab library based on its sub-library and sub-table can realize the internal integration of most of the technical points mentioned above. For users, they may only need to initialize the Client, and then write SQL normally. The final result can be directly aggregated and returned, including the intermediate routing strategy, which is implemented by the Client itself through the unified configuration center. This process will involve more support for multiple languages. Relatively speaking, the advantage is that there will be fewer catastrophic consequences when the middleware hangs or goes down. In addition to this, recently, considering that there may be more than one or two language stacks for the entire company level, and not all businesses need to be implemented using Client, a more friendly sub-database and sub-table middleware should be provided at this time. , so later the Inke technical team investigated the form of Atlas Proxy, and also internally developed an Atlas Proxy version that supports sub-database and sub-table.
Key technical point - distributed transaction
The third key technical point is distributed transactions. In the past, all businesses could be solved by a local transaction on a database. Now, it has been divided into many library tables and many dimensional modules according to horizontal splitting and vertical splitting. At this time, it involves the management of data consistency. In this regard, from the business level, there are two types of distributed transactions that may be processed by the business. The first type is the resource preemption type, and the preemption resource type is, for example, A wants to give gifts to B or A to B When transferring money, A needs to deduct money. This part needs to be deducted in advance and then added to B. This process is resource-occupying for A. What are the characteristics of preemptive resource type? In fact, if you want to add money to B, you must ensure that A's money has been deducted, and if this operation does not succeed in the end, it is very easy for A to roll back. The requirements for distributed transactions that are equivalent to preempting resources are real-time, synchronous, and guaranteed to be rolled back. This part of the model that is currently used in the industry is based on compensation, that is, the distributed transaction framework of tcc. This framework model is also widely used in e-commerce and financial-related businesses.
The second type of distributed transaction is given to resources, or the example of the transfer just now. If A transfers money to B, use the resource-giving transaction. If the transaction fails, but B has already transferred the money, then I think It is impossible to roll back. Therefore, for resource-based distributed transactions, either do not add money to B first, that is, by freezing resources, or ensure certain success through some asynchronous and must-reach methods. The characteristic of resource-based transactions is that their real-time requirements are not as strong as those of resource-based transactions, and a certain delay can often be allowed. For example, when A transfers money to B, it does not have much impact on the late arrival of B's money. Therefore, given the characteristics of resource-based transactions, it is more suitable to implement some asynchronous operations and ensure their success.
The above are the problems faced by the Inke technical team in the distributed transaction part and what they have done after realizing the evolution of the database architecture.
4. Analysis of typical application scenarios of live broadcast
It is difficult to say that any architecture that is separated from the actual application scenario is a good architecture, so it is still necessary to analyze the application scenario of the architecture, and analyze what kind of architecture model should be selected according to its application scenario. Next, I will share with you the technical points related to the database and how to design and iterate the database architecture under the application scenario of live broadcast.
The picture below shows a live broadcast big V. The characteristic of live broadcast big V is that it has a lot of fans. The big V in the picture below has 1.56 million fans, and the largest number of fans may reach tens of millions. The problem caused by the large number of fans is that whether it is the amount of push, attention, or activity in the live broadcast room, it will be very high. And in itself, the live broadcast of big V will be more frequent. At this time, the biggest test for the platform is the relational system, so the relational system must be able to support high concurrency and large capacity, and the requirements for the real-time and consistency of the relational system are also relatively high. If A follows B, but fails to follow in the end or does not receive the push information from B's live broadcast, then this is a fault. So this puts a lot of pressure on the relational system, because the mutual relation that can be established between any user is itself a form of Cartesian product.
Therefore, the Inke technical team has also optimized the relational system, especially the relational database. The optimization of relational databases is mainly divided into several levels. The first is read-write analysis, that is, the writing and reading of this part of the relational database are separated; in addition, the asynchronous writing is realized, and each relationship in this part is written. There will be a lot of write operations involved in the triggering of the trigger. At this time, it should be ensured that after the write operation of the critical path is successful, everything else is done in the form of writing asynchronous; Breaking through the limitation of a single database instance, it is necessary to do a lot of work on data splitting such as sub-database and sub-tables; in addition, there is a feature in the relational system, that is, the message push needs to involve the relationship, and the two people chatting can also Relationships need to be involved, and each scenario that requires relationships is different, so a lot of heterogeneous data processing work needs to be done.
As you can see from the figure below, there are many asynchronous optimizations on the main relational service of the relational system. First of all, in order to achieve these optimizations, it is necessary to ensure that the relational service itself and the relational data itself are gathered and have a unified entrance. After that, you can control this entry uniformly, and then ensure that the critical path is successfully written, and other non-critical paths can be distributed through the message queue, and some heterogeneous data sources or different types need to be optimized for cache and different. Structure data construction, etc.
As you can see from the above picture, if the anchor has 10 million fans, then every time the anchor starts broadcasting, he needs to check and push all 10 million fans, then the pressure on the system will be greatly reduced. is very large. So in this process, through Redis, some data splitting ensures that even if the anchor has 10 million fans, it can achieve second-level push. In some scenarios, such as watching live broadcasts, if the anchor has 10 million followers, if the data of these 10 million followers is written every time the broadcast starts, it will cause a lot of pressure on the relational database. In fact, You can use the watch list to check all the viewers who are watching the live broadcast and push them, so that these points that have a great impact on the system performance can be solved in a heterogeneous way through the combination of push and pull.
The picture below also shows a big anchor on the Inke platform. In the daily rankings, this anchor has received more than 40 million tickets in one day, and one ticket is equivalent to a dime. Each user can send a small gift, a screening ticket, and can send in a row. This will cause this problem, because the more than 40 million tickets are sent to the anchor within one day, or even within an hour or half an hour, which will inevitably cause a lot of pressure on the database. The data is only one piece of data in the database, so many update operations will be involved. Therefore, based on such a high-concurrency scenario of gift-giving, Inke has also optimized many levels. First, it supports the strong consistency in the financial attributes of the gift-giving scene. Although a screening ticket is only 10 cents, the value of the gift is small but still high. A financial attribute, if this dime does not go to the anchor's account, it will become a financial failure, and its impact will be very large. And no matter when the fans give the anchor a ticket, they need to see that the anchor's number of tickets is increasing in real time. If the anchor's number of tickets does not increase in a short period of time, the user may come to complain, so in the The requirements for real-time performance in this scenario are also very high. In addition to the need to ensure strong consistency and real-time financial data, it is also necessary to ensure high concurrency when multiple fans send gifts to the anchor at the same time, which puts a lot of pressure on the gold coin system.
Inke Live Gold Coin System
The initial architecture of Inke's gold coin system is shown in the figure below. It is also a MySQL cluster that relies on RDS. It was a single host at the beginning, and then an independent instance was optimized. After that, the read-write separation was performed to realize After the separation of reading and writing, in fact, the business including gift giving, payment and manifestation can still directly operate the database. At this time, even if the gold coin database can support a large amount and a high stability, when encountering the situation just mentioned that the anchor receives 40 million tickets a day, the delay caused by the database is unacceptable.
Therefore, the Inke technical team has also optimized version 2.0 of the gold coin system. Similar to the relationship service mentioned above, the premise of any data-level optimization is to first collect the entrance, that is, let the gold coin system be responsible for all the Operations related to coin data. After the entrance is closed, various optimizations can be carried out. The first level is to do the work of sub-database and table, and the second level is to optimize the process of the host receiving gifts asynchronously. Why implement these two levels? Perhaps sub-database and sub-table cannot solve the pressure of an anchor receiving more than 40 million tickets a day, but it actually solves another problem, which is to improve the overall concurrent throughput. It may be that the gifts sent by the audience will all converge on this anchor, but for the audience, each user is a separate individual, and for them, the operation is not particularly fast, and the audience is discretely distributed in On different databases, at this time, the data can be split by sub-database and sub-table to minimize the pressure on the audience side. On the other hand, asynchronous optimization can be used. If there are more gifts on the host side, asynchronous serial processing or even combined processing can be used to ensure that the pressure of the system is within a controllable range. The premise of these two layers of optimization is that the entrance must be closed, which is Inke's consideration for the optimization of the gold coin system.
Gift-giving logic analysis
Here I share with you a core gift-giving logic in business. It was also mentioned before that the gift-giving operation and the gift-receiving operation need to be divided into two parts, and each part needs to adopt different strategies, one is the strategy of sub-database and sub-table, and the other is solved by using the asynchronous strategy.
The distributed transactions just mentioned include preempting resource transactions and giving resource transactions. Gift-giving is precisely a resource-based transaction, which must ensure that gift-giving has a better real-time performance and can perform synchronous operations. Because gift-giving involves synchronous operations, it needs to be real-time, and the audience's relatively discrete operations during gift-giving can easily ensure an overall throughput improvement by sub-database and sub-table. The synchronization operation here is not a simple update operation. Once the update operation fails, it is difficult to achieve fault tolerance. This involves logging of local transactions for each operation itself, which can be documented if it fails or causes some idempotency problems. In addition, on the gift receiving side, this is equivalent to the operation of giving resources in distributed transactions. Since resources are given and a certain delay can be tolerated, the transaction can be put into a message queue after the deduction is successful. The gift-receiving operation is processed asynchronously through the message queue, and channels can be isolated for large and small hosts, so that the needs of the gift-receiver are well met. In general, such a logical design ensures the stability, high concurrency and real-time performance of both the gift-giving and gift-receiving parties.
Summarize
In fact, the database itself is an infrastructure. Then as an infrastructure, we can mainly rely on the stability and basic performance of the database. The cloud provides expert-level operation and maintenance for the database, which is also a good aspect. However, cloud databases are not omnipotent. In many cases in the aforementioned scenarios, specific architecture must be sorted out and optimized at the architecture level and at the specific business scenario level. In terms of database architecture, we have been on the road, and will continue to work hard to move forward.
This article is the original content of Yunqi Community, and may not be reproduced without permission. If you need to reprint, please send an email to [email protected]