Architecture Design and Evolution of iQIYI Platform

In recent years, iQIYI has developed rapidly, and high-quality content has emerged in an endless stream. iQIYI’s advertising has also developed and expanded. Online advertising services also serve different customers such as brands, small and medium-sized enterprises, and DSPs, forming a relatively complete business that can meet different types of needs. Advertising monetization layout, advertising inventory covers video, information flow, bubble social (iQIYI's social platform) and startup screen and other scenarios. iQIYI Effect Advertising is a brand-new advertising delivery platform established in 2015. With the growth of the information flow business, the entire delivery platform has also undergone a major structural adjustment and several important upgrades and optimizations.

The general architecture of iQIYI's advertising platform is shown in the figure below. This article mainly introduces the content related to online services. The online delivery service is the part framed by the dotted line in the figure, which mainly includes online delivery and billing services.

The business requirements behind the architecture

The architecture is definitely born for business needs, let's take a look at the business needs and characteristics we are facing.

The iQIYI effect advertising platform currently adopts the agency model. The platform mainly meets two types of business needs: oriented to agents (advertisers) and oriented to the needs of products and operation teams. Let's take a look.

1. The needs of agents: In essence, it is to help agents reduce conversion costs

• Support a variety of advertising spaces: patch, pause, floating layer, information flow, video related position and recommendation position, etc.

• Support multiple settlement types: support CPC, CPM and CPV and other advertising settlement types, oCPC settlement method is planned

• Rich targeting functions: common targeting dimensions (platform, region, etc.) and precise targeting of crowds (regional targeting - supports district and county levels, crowd attribute targeting and DMP crowd targeting), keyword targeting

• Flexible scheduling and budget settings: support minute-level scheduling settings, support arbitrary increase or decrease of daily budget

• Special business functions: advertising deduplication, dynamic creativity, creative optimization and smooth consumption, etc., are all to improve the conversion effect of advertisements

• Frequency Control: Avoid a large number of exposures to the same user in a short period of time

2. For the product and operation team: mainly to improve the product control ability and promote the good operation of the overall system

• Traffic control: Control which ads cannot/can be served on certain traffic through black and white lists

• AB testing function: AB testing is required before the full release of functions with greater impact to confirm that the effect is as expected

• Billing related: delayed exposure is not billed, exposure, click anomaly detection and filtering

• Negative feedback: Automatically adjust advertising strategies based on user feedback to optimize user experience, which is also a constraint on advertisers

From the business requirements described above, it can be seen that the characteristics of the business are:

1. Complex business logic: The process includes many steps (scenario information acquisition, advertisement recall, budget control, frequency control, CTR estimation, creative optimization, smooth consumption, advertisement deduplication, result sorting, result screening, probability delivery, AB testing); The green box in the figure below only shows the main process of serving:

2. Business changes are very fast: an average of 5 system function changes per week;

1.  

2. The number of advertisers is large, the order volume is large, the average order budget is small, and the order settings change frequently.

system structure

The iQIYI effect advertisement was officially launched in 2016. At the beginning, the business logic was simple, the number of advertisements and orders was small, and the overall structure was relatively simple. In order to quickly complete the construction and online application of the system, the architecture of the brand advertising platform is reused and tailored. The system architecture diagram is as follows:

The access layer  includes QLB (iQiYi Load Balance) and Nginx front-end machines, which are mainly used for reverse proxying of traffic and overall current limiting and downgrading functions.

Traffic distribution layer : including strategy services and traffic platform services; strategy services support company-level strategy control and daily operational needs; traffic platform services mainly control traffic distribution and request logic on various delivery platforms, including brand advertising platforms. , performance advertising platform and external DSP.

Delivery service : The business logic described above is included here and implemented by a single module.

Log collection : Receive logs such as exposure clicks, and mainly complete business logic such as billing, frequency control, and deduplication, which is also implemented by a single module.

Billing system : Use the Redis master-slave synchronization mechanism to synchronize the real-time consumption data of the order to the delivery service.

Frequency system : use Couchbase cluster for user data storage.

Data synchronization layer : There are many types of data involved in this layer, two of which are relatively important: business data and log data. Business data mainly includes advertising orientation, scheduling, and budget.

We use business data to optimize two aspects:

1. Distribute some data that does not require high timeliness to the delivery service through business data, avoiding some network IO;

2. Perform space-to-time optimization in business data, including index generation and pre-computing of some data required for delivery services, such as pre-computing key values ​​in the billing system.

As the business grew, the architecture also encountered some challenges.

1. Traffic growth: After the system was launched, it has well met the requirements of advertisers for conversion effect. This positive effect stimulated advertisers' demand for traffic. For this reason, the product and operation team continued to open up new advertising spaces. At the same time, iQI The number of users and traffic of Yiyi has also continued to grow, and these reasons together have brought huge traffic to the performance advertising platform.

2. Growth in the number of advertisers and orders: This growth includes two aspects. On the one hand, it complements and promotes traffic growth; iQIYI's high-quality traffic and good conversion effects attract more advertisers; on the other hand, due to business policies For the above reasons, advertisers and order volume will have a phased growth at the end of the quarter.

3. Performance problem: The increase in traffic and order volume makes the system load increase rapidly, because orders are recalled in full. When the order volume increases to a certain number, it will increase long-tail requests and affect the overall service performance, which cannot be solved by horizontal expansion.

4. Over-investment problem: Due to the delay of exposure and clicks, as well as the delay of the delivery billing loop, there is an inevitable over-investment problem, which is also an inherent problem of the advertising system; brand advertising is to sign a contract first, and the delivery volume can be followed according to the contract. Collection, the excess part will not be charged to the advertiser, brand advertisements have a large amount of reservations, and the over-investment ratio is small; different from brand advertisements, the effect advertisements are deducted in real time. If the brand idea is followed, the over-investment part will cause excess Small and medium-sized advertisers are very sensitive to this, and will also increase the technical team’s problem analysis and investigation work. At the same time, because the budget of performance advertising is small, the budget adjustment changes rapidly, making the over-investment ratio larger than that of brand advertising; The over-investment problem, the technical team has to do is divided into two levels, one is to ensure that the over-investment part will not be billed, and will not bring losses to advertisers, and the second is to fundamentally reduce over-investment, that is, reduce our own income. Losses; respectively referred to as  over-investment without billing  and  over-investment reduction ;

In response to the above situations, our architecture has been adjusted:

Compared with the structure at the beginning of the launch, the structure adjustment at this stage is reflected in the following aspects:

1. Delivery service performance optimization – including index sharding and adding a coarse sorting module, which mainly solve the performance problems caused by the above-mentioned traffic growth, advertiser number and order growth, etc.

2. Index sharding is to split the original index into multiple copies, and a corresponding request will be split into multiple sub-requests for parallel processing, so that the processing time of each request will be reduced, thereby effectively reducing the number of long-tail requests.

3. Coarse sorting: The advantage of full recall is to maximize revenue, but the disadvantage is that performance will decrease linearly with the increase in order volume; coarse sorting filters out uncompetitive low-value (low ECPM) advertisements in the recall stage, and low-value advertisements The probability of being placed and the probability of generating conversions is very low, so the filtering of coarse sorting has little impact on the overall revenue, and at the same time, it can effectively reduce the number of orders entering the subsequent core computing logic (including fine sorting and other business logic), making service pressure. Does not grow linearly with order volume.

4. Billing service architecture optimization - mainly to improve the scalability of the system and solve the problem of over-investment

Scalability is solved by service splitting. The billing service of a single module is split into three modules. After splitting, the log collection module provides external services. The main responsibility is to receive log requests and return them immediately to ensure extremely low response time. ; Then process the billing log and non-billing log differently; the main responsibility of the detection and filtering module is to perform directional inspection and abnormal log identification. The billing service updates the valid billing data to the billing system. After being split into three modules, each module is simple and conforms to one of the fundamental principles of microservices: single responsibility .

Regarding over-investment, let’s first look at the first question: over-investment is not billed.

The main difficulty is:

1. Billing requests for the same advertisement are concurrent;

2. The billing system is distributed, and for performance reasons, the request processing flow needs to be lock-free.

The way we solve this problem in the billing system is as follows:

First of all, in order to charge strictly and accurately, it is necessary to serially process parallel requests. The single-threaded model of Redis naturally meets the needs of serial billing. We decided to implement this architecture based on Redis, and put the billing logic in scripts. The form is executed in the Redis thread, which avoids the logic of reading first and then writing, so that the two root causes are eliminated.

The next task is to design a high-availability and high-performance architecture based on Redis. We considered two alternatives.

Option 1: Data fragmentation, there are multiple master Redis in the architecture, each master Redis stores a fractional fragment, and the log collection service needs to update the master Redis when processing valid billing requests; each master Redis has a corresponding read-only slave Redis, the delivery service obtains the corresponding real-time consumption data of advertisements from Redis according to the sharding algorithm.

The advantage of this solution is that it has strong scalability and can be expanded to solve performance problems; the disadvantage is that the operation and maintenance are complex, and it is even more complicated to meet the high-availability system architecture;

Option 2: The data is not fragmented, all billing requests are aggregated to the only main Redis, and the read-only from Redis can sink to the delivery service node, which can reduce network IO and make the architecture more concise; but the main Redis can easily become a performance bottleneck;

In practice we adopt the second  non-sharding  scheme. Mainly based on the following considerations:

At the business level, a large percentage of effect advertisements are CPC advertisements, and the number of click logs is relatively small, which basically does not bring performance pressure to the system; for the remaining CPM-billed advertisements, the system will charge The log is aggregated to reduce the pressure on the main Redis; because the redis is sinking to the delivery, no special high-availability design is required; the high-availability of the main Redis adopts the Redis Sentinel scheme to achieve automatic master-slave switching, log collection The service gets the latest master Redis node through the Sentinel interface.

In the case of serial billing, the last billing request may still exceed the budget after the accumulation. Here is a small optimization technique, adjusting the actual billing value of the last billing request to make the consumption exactly match the budget.

The second problem about over-  delivery is to reduce over-delivery. This problem cannot be completely solved, but it can be alleviated, that is, to reduce the ratio of over-delivery without billing, and to minimize inventory loss; our solution is to charge for advertising When the consumption is close to the advertising budget, probabilistic delivery is performed. The closer the consumption is to the budget, the lower the probability of delivery; this method has a disadvantage that it does not take into account the differences of advertisements. Some advertisements have low ECPM and their own delivery probability is very small. The impact of exposure (or click) delay is also very small; for this, we have made another optimization: based on historical data to estimate the advertising budget consumption speed and billing delay, and then use these two data to correct the delivery probability value.

The biggest feature of this solution is that it is simple to implement. It can be implemented by simple development in the existing system, without adding additional system support, and does not depend on accurate business scenario prediction (such as exposure rate, click rate, etc.), and The results are not bad; we are still trying different ways to continue to optimize the overinvestment ratio, because as the income grows day by day, the income loss caused by the overinvestment is still considerable.

Thoughts on Microservice Architecture Transformation

The microservice architecture has now been widely accepted and promoted in the industry, and we have a strong sense of identity with this architectural idea from the very beginning; Advertising Online Services completed the construction of the first version of the main architecture in 2014. At that time, the microarchitecture ( The dotted box represents a server) like this:

Deploy multiple services on the same machine. The upstream service only requests the downstream service of the machine. The HTTP protocol is used to transmit protobuf data between the services. Each machine is a complete delivery system.

This architecture has many advantages: clear structure, simple operation and maintenance, minimal network delay, etc.

Of course, there are also some disadvantages. The number of services that can be deployed on the same machine is limited, which will limit the growth of the architecture. The mixed deployment of multiple modules is not conducive to the overall performance optimization. The abnormality of one service will affect the service quality of the entire machine; this The architecture satisfies the principle of a single service at the micro level, but it is not a real micro-service at the macro level. In order to solve some of the above problems, according to the natural evolution, we will inevitably embark on the road of micro-service; Start the practice of microservices.

We have also encountered many problems in the process of microservices, and share our solutions and effects:

1. Technical selection problem

RPC selection, the conditions that must be met are to support C++, protobuf protocol and asynchronous programming model. The initial options are sofa-pbrpc, pbrpc and grpc. Among these three, we chose grpc, mainly focusing on its general features (multi-language, multi-platform and support for proxies), flow control, cancellation and timeout; in our Shortly after selecting grpc, Baidu open sourced its high-performance rpc framework brpc. Compared with brpc, brpc has more advantages: sound documentation, high performance, built-in detection services and many other features; for this reason, we decisively abandoned grpc and brpc. Some development costs that have been invested in the above have quickly launched the development of basic functions related to brpc and the transformation of various services.

Name service selection, excluding zookeeper, etcd, etc., finally selected the combination of consul+consul template, which perfectly supports our business needs; in addition to service registration and discovery, there are also health checks, service list local Backup, support weight setting and other functions, these functions can effectively reduce the operation and maintenance workload of team members, enhance the availability of the system, and become the standard configuration of the service.

2. Increased operation and maintenance costs

This is one of the problems brought by microservices. In microservices, services need to be split, the type and number of service nodes will increase, and some additional basic services (such as the agency of the name service) must be operated and maintained. Considering that most of the operation and maintenance work is repeated execution of the same task on multiple machines, such a problem is most suitable for the machine to complete, so our solution is automated operation and maintenance. We developed a visual automatic operation and maintenance system based on Ansible. In fact, the original purpose of developing this system is not to support microservices, but to eliminate manual operation and maintenance accidents, because the state of people is unstable (sometimes even unreliable), so it is hoped that machines will replace people to complete Repeated standard actions; later, with the advancement of microservices, this system naturally took over the related operation and maintenance work. Now this system has completed more than 90% of the operation and maintenance workload of the entire team.

Automatic operation and maintenance system architecture

1. Problem discovery and analysis and positioning

The common methods in the industry are full-link tracking system (dapper & zipkip) and intelligent operation and maintenance, and we are also working on this aspect; in addition, we have also done two other things: anomaly detection and staging environment construction ;

• Anomaly detection: It mainly finds anomalies in various macro indicators from the business level. For the advertising delivery system, inventory, exposure, click rate and billing rate, etc. are all very concerned business indicators; the anomaly detection system can predict business indicators. The reasonable range value at the current moment is then compared with the real-time data; if the real-time data exceeds the predicted range, an alarm will be issued and the analysis data will be attached to assist in the problem analysis; this part of the work is completed by the online service and data team, this system effectively Improve the efficiency of problem discovery.

• Staging environment construction: System changes (including operation and maintenance and new function release) are the main reasons for online failures, so we need a system to help us quickly find abnormal changes at a small cost.

•  

When a function is released, everyone will adopt the method of gradient release. For example, upgrade 5% of the services first, and then observe the changes in the core indicators. If there is no obvious abnormality, continue to advance until the full amount; this method is not always effective in finding problems. If a new function The bug in 1% will cause the exposure of 1% of the orders to drop by 50%, then the overall exposure of the system will only change by 0.5% after the full release. It is also possible that the overall exposure does not change due to the filling of other orders, so only through the overall exposure It's hard to spot this problem. Only a comparative analysis of the exposure of all orders can accurately detect this problem.

In practice, we use vector cosine similarity to find anomalies caused by system changes, that is, convert the number of advertisements exposed within a period of time (5min) into vectors and calculate cosine similarity. So how to get two vectors? A vector can be generated before and after the division according to the time of the gradient release. This method is not robust enough, and the vectors at different times have certain differences.

We solved it in this way: deploy an independent delivery environment (we call it Staging environment, relative to the original delivery environment called Base environment) to carry a small amount of online traffic (such as 3%), and all system changes are made first Do it here; then use the vector of the Staging environment and the vector of the Base environment to calculate the similarity.

Because it is very sensitive to differences, false positives will occur when using cosine similarity for monitoring; but this is not difficult to solve. Through some bad case analysis, we locate and eliminate the differences (non-bug) factors between the two environments; Under normal circumstances, the similarity between the two environments will remain at about 95%, and when a real anomaly is encountered, there will be a significant drop to trigger an alarm. Staging environment and similarity detection function have helped us find system anomalies many times in practice.

Experience accumulated during the architectural design process

Finally, I would like to share some experiences that I have summarized during the architectural design process.

• Deep understanding of business.  In terms of architecture design, business and architecture need to cooperate with each other. While the architecture meets business needs, it can also in turn raise demands for business and even require changes to business logic to achieve the optimal system. The key here is to fully understand the business. Problems that are difficult to solve in terms of structure may be solved with a small adjustment in the business. If it can have such an effect, why not do it. In terms of system or architecture optimization, optimization theories and strategies have been fully researched, and the only thing left is how to combine and utilize them with business scenarios.

• Perfection should be pursued in the design stage, cost-effectiveness should be considered in the practical stage , and a perfect architecture should be evolved in a step-by-step manner. ** In the design stage, you can temporarily put aside the constraints of the implementation cost or some other objective conditions, and do the architectural design according to the ideal situation. The result obtained in this way is an ideal goal that we are pursuing. It does not matter if this goal is not achieved temporarily. Because its function is to indicate the general direction of the future development or evolution of the architecture; and then gradually adjust the perfect architecture to a practical level in combination with actual constraints. In this process, multiple intermediate versions can be retained as the architecture. Milestone for the evolution upgrade process. It can also be understood in this way, starting from reality and focusing on the future, as the speed of technological development is getting faster and faster, the restrictions and obstacles encountered at the beginning of the design will soon be solved, so as to avoid being blocked by these temporary restrictions and obstacles. Live the imagination of the future.

• Monitor first.  Monitoring information is important information to understand the operating status of the system. Most of the monitoring information must be persisted for data analysis. It can be used for anomaly detection and can also assist in problem analysis and localization; doing a good job in monitoring is to improve TTA (Time To Detection) and TTM (Time To Mitigation) indicators; it should also be emphasized here that the relevant various monitoring indicators, statistical granularity and other details should be considered in the design stage, and carried out in the system in the development stage. Calculation and statistics of relevant indicators, synchronize these indicators to the monitoring system during the service deployment stage; ensure that there is corresponding monitoring "escort" at the beginning of the service launch to avoid streaking.

• fault tolerance.  The world is imperfect, and systems in an imperfect world have to face various problems; in the entire life cycle of a system, R&D, operation and maintenance personnel spend a lot of time dealing with and solving various errors and even Disaster; two aspects to consider, namely Design By Failure and disaster rehearsal (Netflex has open sourced their related tools). I would like to talk about my actual experience:

First of all, at the beginning of the design, the boundary of the system can be demarcated, and the inside of the system and the outside of the system can be separated; from the perspective of cost, because of the strong controllability inside the system, some assumptions can be set to reduce related considerations and system fault-tolerant design ; For the rest of the internal system and external system problems, priority should be given to solving problems with greater impact (for example, unavailability of external services, sudden increase in external interface access) and high-frequency problems (hard disk failure, network cutover), so that Most of the problems in the network have solutions that can be used for reference. If the existing solutions cannot be reused due to special business scenarios, then you must consider implementing them yourself; in response to the unavailability of external services, fuse and increase the bottom guarantee strategy, and limit the sudden increase in traffic. Streaming, when the dedicated line fails, go to the external network, and the hard disk is Raid; other problems are not considered. When the problem occurs for the first time, it is necessary to evaluate the loss and response cost to decide whether to solve it immediately;

Secondly, the idea of ​​disaster drills is the same as fire drills. On the one hand, fire drills can find defects in the escape process, and more importantly, they can cultivate participants' common sense and practical experience of escape, so that they can respond correctly when problems do occur; The same is true for disaster drills. While doing automation, special personnel (especially newcomers) should be arranged to deal with faults, and experienced drivers should be accompanied to review and guide or take over processing actions when necessary. Only in this way will the overall fault-tolerant emergency handling capability of the team be continuously improved. The world is destined to be imperfect, so there are and need perfectionists to make the world perfect, even just a little bit.

Here I recommend an architecture learning exchange group to everyone. Communication and learning group number: 744642380, which will share some videos recorded by senior architects: Spring, MyBatis, Netty source code analysis, high concurrency, high performance, distributed, microservice architecture principles, JVM performance optimization, distributed architecture Wait for these to become the necessary knowledge system for architects. You can also receive free learning resources, which are currently benefiting