Hello everyone, my name is Chen Peng and I come from Tencent.
The first three teachers spoke in depth and were full of dry goods. Compared with the depth of the first three teachers, the topic I shared today will be lighter. I will mainly introduce you to the exploration of Tencent’s focus on the delivery system. There is no academic and profound principles, and no catalog. There are only stories here. Follow me Let’s start a story with the PPT.
Before introducing the delivery system, I would like to make a simple survey. How many people have used information streaming products? Such as Tencent Kandian, Today's Toutiao and so on. Quite a lot, then ask again, when browsing the information stream, how many people have seen content that disgusts them? I found that the students who just raised their hands basically also raised their hands. This shows that low-quality content in the information flow is still quite common, so how to solve it? This is actually a difficult thing. Our information flow students have been exploring, and the delivery system is one of the products of exploration.
Speaking of the delivery system, the recommendation system cannot be bypassed, because the delivery system was born to solve the problems of the recommendation system. So what is a recommendation system? Teacher Pan has just given a very detailed and in-depth explanation, I will simplify it here.
Imagine, now that a user opens the information flow product, what will happen next? The front-end will request the back-end, and the back-end will request the recommendation system, and the recommendation system will go to the content pool to find a few pieces of content that the user is most likely to like. How does the recommendation system know what type of content users like? User portraits, each user will have some tags on them to indicate what they like or hate. For example, the portrait of this user shows that he prefers King Glory, Finance, Sports, etc. Each piece of content also has its own label, indicating what kind of content it is. What the recommendation system needs to do is to match the user with the content. For example, if there is "Honor of Kings" in the portrait of this user, the recommendation system will recommend the contents of the King of Honor.
To summarize, what the recommendation system does is to match user information and content information with each other, and then recommend the matching result to the user, which is the recommendation result. Is there any problem with this? What if this is a new user and there is no portrait? The recommendation system has a content pool, which is specially prepared for these new users. Because the content in it is filtered by the machine, there is no shortage of headline and sideball content. When users see this type of content, although their brows are frowned, they are still I often can’t help but click. It doesn’t matter if I click, that’s it. If there is a big misunderstanding, the recommendation system will think that this user likes this kind of content, and then it will try to push similar content to the user. The user is disgusted, but I can't help but not watch it, so it will form a vicious circle. When users lose patience, they won't use this product again.
How to break this vicious circle? One way is human intervention. The machine may be stupid to push you content that seems popular and actually of lower quality, but our operating students will only select high-quality content, and then push it to users along with the results from the recommendation system. So can users see high-quality content?
Now let’s look at the process of requesting data. When a user accesses an information flow product, the front-end will request the back-end. The difference is that the back-end will request the mixed-ranking layer, and the mixed-ranking layer will request the recommendation result from the recommendation system, such as The recommendation system recommends two content related to the glory of the king; on the other hand, the mixed ranking layer will inquire whether the operating students have manually recommended content, for example, the operating students here manually recommended this content related to "Thirty Only". After the mixing layer receives the content of the recommendation system and the operation configuration, it mixes and arranges the data on both sides according to certain rules, and then returns the data to the front-end users together. In this way, users can see high-quality content selected by recommended classmates.
We continue to ask, will there be any problems with this? The content is good content, but not everyone likes it. For example, as I just said, the operating students recommended the content of "Thirty Only", but in fact, quite a few users don't like it. Every exposure opportunity is very important, especially the first few. If users don’t like "Thirty Only", they will not click, which will inevitably lead to a decline in market data. What should I do? If we can improve the accuracy of operation push and push different high-quality content to users who are likely to be interested, will it not only solve the problem of large-cap data decline, but also solve the problem of pushing high-quality content to users? What?
How to improve the accuracy of operational push? The secret is our protagonist today: the delivery system. The work done by the delivery system can be summed up in one sentence, which is to deliver as accurately as possible the content that the operating students consider high-quality to users who may be interested. There are two main points here. First, what content will be delivered? Second, who is the content delivered to? Look at it separately.
The first point is what content to put. The content delivered by the delivery system is also the content that enters the content pool after warehousing services. Our operating students will select high-quality content from the content pool and classify them. The operating students will aggregate these high-quality content together according to their needs. These aggregated content according to certain rules are called "points of interest". The rules here can refer to the same account, the same category, the same topic, and the same topic. Wait a minute, as long as our operating students think there are similarities, they can aggregate these contents together. Let’s give some examples. The famous singer Hu Xia is an account. The category is like food, the theme is like Marvel, and the topic is like "Thirty Only", where "Hu Xia", "Food", "Marvel" and "Thirty Only" are points of interest. We can see that there are actually no rules for the division of interest points, the granularity can be large or small, and the dimensions are flexible and diverse. To summarize, the content delivered by the delivery system is based on points of interest, which are an abstract concept and represent the same type of content.
After talking about the first point, let's go on to the second point: Who is the content for? We just said that content is aggregated according to certain similarities, and the job of the delivery system is to deliver this type of content to users who may be interested. Having said that, we can better see the benefits of aggregating content together. Finding target users for a piece of content is very difficult, but finding target users for one type of content is much easier.
Speaking of who the content should be delivered to, the delivery system actually didn’t know at the beginning, but it will continue to explore and find the target consumers. For each point of interest, the delivery system will tentatively deliver the content in this point of interest to some users. Take the point of interest "Thirty Only" as an example. Assume that the delivery system will deliver about 5 articles to a group of users every day. That’s it. Some users are not interested in this topic, and may not click on one of them. Some users, such as I know this topic, hold a neutral attitude, and may click one or two articles to read. Is a loyal fan of "Thirty Only", it is likely that four or five will be featured. The delivery system has a set of algorithms that will score users according to their clicks, and divide users into three categories: non-audience, related users, and core users. This is the situation on the first day. The delivery system will circle a part of users for each point of interest for exploration, and then divide users into three categories of non-audience, related users, and core users of this point of interest based on the exploration results. Then, on the second and third days, continue to explore, can you circle more and more relevant users and core users of this point of interest in the vast crowd? Indeed, these two types of users are important assets for this point of interest.
When a point of interest is just created, it does not have any related users or core users, then we need to explore which users may be interested in this point of interest. How to explore? One method is to explore randomly, select some users at random, and then deliver content to them. There is no doubt that this is certainly inefficient. Is there a more efficient way? Here are three methods to introduce to you, namely: manual exploration, machine exploration and strategy matching.
Let's first look at manual exploration. Whether you have played it or not, I believe many people should know about the game Dungeons and Warriors. Recently, the mobile game of Dungeons and Warriors will be launched, but we will not talk about the game, only the content derived from the game. The launch of the new mobile game will definitely produce a lot of relevant high-quality content. Our operating students keenly grasped this point and created a point of interest called "Dungeon and Warrior Mobile Games" in the launch system in advance. The question is, the content is there, what about the target users? Don’t panic, the professional experience of our operating classmates will come in handy. They will set several screening criteria, such as gender, age, income, etc., and then screen out users who are most likely to like "Dungeon and Warrior Mobile Games" For example, choose male gender, choose age between 15-35 years old, choose first-tier cities by region, choose unlimited operating system, choose monthly salary above 8000 for income level, and choose game animation category for user label. These conditions have greatly improved the accuracy of exploration, which is the basic principle of manual exploration.
How do our operating students know the user’s gender and age? User portrait. Currently, there are two types of portraits we use. One is Tencent's user portraits, such as finance, animation, fashion, etc., and the other is user portraits for other businesses. For example, IEG game portrait, Tencent video portrait, QQ music portrait, etc. To sum up, manual exploration is a way of launching new users to explore points of interest that relies on professional experience in operations.
After talking about manual exploration, we continue to look at machine exploration. We just mentioned that when a point of interest is just created, there are no core users or related users. After some users are selected and explored by operating students, this point of interest will precipitate some core users and related users. We use small blue People said. The so-called grouping of people by category, the users represented in blue must have some common characteristics, we extract these common characteristics, and then find users with these characteristics in the market, and then filter them out as ours Explore users. These exploration users are screened out by machine algorithms, so they are called machine exploration. To summarize, machine exploration is a way of launching new users to explore points of interest that relies on machine algorithms.
The above is machine exploration, we continue to look at strategy matching. Suppose a user searches for "Dungeons and Warriors" in QQ. Do we naturally think that this user is interested in "Dungeons and Warriors"? Then this user did his part to become an exploration user of the point of interest of "Dungeon and Warrior Mobile Games". When this user searched, we triggered a specific strategy: "Search for dungeons and warriors", where search is behavior, dungeons and warriors are keywords, and searching for a certain keyword is one of our strategies. Strategies are actions that trigger specific signals. At present, we have set up six strategies. Of course, we have been exploring other strategies. Some strategies may be set for each point of interest. If these strategies are triggered, they will become explorers of this point of interest.
The above are the three exploration strategies we use: manual exploration, machine exploration and strategy matching.
For each point of interest, the delivery system will explore a group of users who may be interested in this point of interest every day, and then deliver a few pieces of content in this point of interest to them. Some users don’t have a point in an article, which is the non-audience of this point of interest; some users occasionally click one or two articles and become related users of this point of interest; and some users may read most of the content and become this Core users of points of interest. Core users and related users are important assets that have precipitated this point of interest. Is this a one-shot deal? Core users will always be core users, and related users will always be related users? Imagine that if 5 articles of a certain point of interest are posted to a user on the first day and the user reads 4, then he will be judged by the system as the core user of this point of interest. In addition to delivering content to exploring users every day, the delivery system will continue to deliver content to core users and related users that have been deposited before. If the content of this point of interest continues to be delivered to this core user on the second and third days, but this user may not click on an article, our system will adjust his score and may reduce him to this Related users of points of interest or even non-audiences. So the user's identity is not static, it is a dynamic process of change.
Recap what was just said. Points of interest are content aggregated according to certain rules. The goal of the delivery system is to find target users for the points of interest. When the point of interest is just created, the delivery system does not know the target users of the point of interest, so it needs to explore. Here are three ways for the delivery system to explore new users: manual exploration, machine exploration, and strategy matching. The so-called exploration is to find users who may be interested in this point of interest, and then deliver content to them to verify how much they like this point of interest. According to the degree, we divide users into core users, relevant users and non-audiences. Our goal is Find more core users and related users.
All this looks perfect, so is there any problem? Suppose that the first point of interest is explored first, and a group of users are explored, the second point of interest is explored, and a group of users are explored, and the third point of interest is explored, and a group of users are explored. Active users watch more content and accumulate more portraits on their bodies. The more likely they are to be judged by the system as exploring users. In this case, many points of interest will be explored by active users, and then they will be declared as core users of this point of interest or Related users. This will cause various points of interest to always be explored around active users, and the exploration of long-tail users is far from enough, which is not conducive to increasing the activity and stickiness of long-tail users.
Is there any way to solve it? Here is a strategy we adopted: setting and exploring the mutual exclusion of users. Now the way of exploration has become, the first point of interest explores some users, the second point of interest explores the core users and related users that are explored by the first point of interest, and the third point of interest explores Need to avoid users who have explored previous points of interest. This forces the following points of interest to shift the exploration range to the long-tail and low-active users as much as possible when exploring users.
We have been talking about the delivery system just now. What is the relationship between the recommendation system and the delivery system?
First of all, there is an essential difference between the two systems. A user opens Tencent Kandian or other information flow products, hoping to recommend several articles of interest to him. This is a process in which the user actively seeks out content. As for the delivery system, the opposite is true. Operating students feel that there are many high-quality long-tail content in the system that have not been recommended. They want to find a suitable user group for these high-quality content, so this is a process of finding people for content.
On the other hand, the delivery system has a profound impact on the recommendation system, we look at it from two aspects. The first aspect is the user dimension. When a user consumes the content delivered by the delivery system, his user portrait will be affected. For example, the delivery system has delivered several articles of "Thirty Only" to the user. After the user consumes it, The portrait on the body will be updated with the label of "Thirty Only". This kind of portrait will be adopted by the recommendation system, so the subsequent recommendation system will also recommend more "Thirty Only" content to this user. The second aspect of the delivery system that affects the recommendation system is the content dimension. For some long-tail unpopular content, if our operations feel high-quality, it will be delivered through the delivery system. If the target audience is more accurate, it will soon If many users read these contents, the recommendation system will consider these contents to be popular content, and then carry out greater promotion.
Okay, let’s go back to the problem we said at the beginning. Because the user clicks on a few low-quality and vulgar content, the recommendation system will mistakenly think that the user is interested in such content, and then continue to recommend similar content to the user. The user may not be really interested. On the contrary, they may be disgusted. They just can't help but click, and then they will fall into a vicious circle. One of the solutions is the delivery system. The operation will select some high-quality long-tail content and deliver them to users through the delivery system, so that the vicious circle can be broken and the user experience and stickiness can be improved.
Speaking of this, the principle of the delivery system is basically finished. It sounds like a lot of benefits, so how to quantify the benefits brought by the delivery system? A tool commonly used by Internet users is A/B testing. In the last few minutes, I want to share with you the mystery behind the A/B test used by the delivery system.
When the product has new features, we will choose two dial users, one dial user uses the latest version, and the other dial user continues to use the old version. After the experiment for a period of time, perform experimental analysis. If the new version works well, the new version will be released, and if the old version works well, the old version will remain. The actual A/B testing process is very complicated. Just now, Mr. Wu shared with us in depth, and I don’t make any trouble. The main thing I want to share with you is the way of selecting samples.
There is a natural user ID in QQ, which is our QQ number. How to select two dial users? One way is to modulo the QQ number, such as modulo 100. Then according to the result of the modulus, the users can be randomly divided into 100 parts. The user with the result of 65 keeps the old version, and the user with the result of 66 uses the new version. The QQ numbers of these two users end with 65 and end with 65 respectively. 66 End. Is this really good? We look at it from two aspects.
First, is it random? Looking at the two dials of users, one dial ends with 65 and the other dial ends with 66. It should be noted that there are nicknames in QQ, and nicknames need to be purchased, so compared with ordinary QQ accounts, the number of nickname users Basic attributes and usage activity are generally good. The two groups of users seem to be randomly assigned, and there should be basically no difference between the two, but in fact, many users ending in 66 are pretty names. This internal difference will affect the results of the experiment, so the modulus is directly taken. It is a pseudo-random way.
Second, is it enough? If the QQ number is 100, only 99 A/B experiments can be performed at most, and there is another group for comparison. Even if it is 1000 or even 10000, it is always limited, so this sampling method is not enough.
Since the modulus of QQ number is pseudo-random and insufficient, how to divide user groups correctly? Here is an introduction to a method we have adopted, which can meet the needs of three aspects at the same time: uniform buckets, enough buckets and no interference with each other. Let's have a brief chat.
The first aspect is even bucketing. We will not directly modulate the QQ number, but first calculate the hash value of the QQ number, and then take the modulus to the hash value. The advantage of this is that the hash function can fully disperse users and avoid the accumulation of nicknames. At the same time, under the size of QQ, we can almost think that the result of modulating the Hash result is uniform. There are situations in which there are many groups of users and few groups.
The second aspect is enough buckets. We just mentioned that we need to hash the QQ number, but this will not prevent the experiment bucket from being used up, so how to deal with it? We can add a salt value. When 99 experiment buckets are used up, we can change to a salt value, which can re-disrupt and group users. It's a bit like playing cards, the first time all the cards will be divided equally into 4 parts, and then reshuffled and divided into 4 parts. Because the salt value is endless, theoretically we can get endless experimental buckets.
The third aspect is not to interfere with each other. Compared with the first two aspects, this is more difficult to understand, let's simplify it. Suppose a product has 81 users, and the A/B experiment is divided into 9 groups. Then there are 9 users in each group. Among them, we number the users in bucket 0, from 0 to 8. Is it possible that after re-distributing users and regrouping, most of the users in the same group entered the same group this time. For example, if the cards were not shuffled, I got one this time Bomb, it is very likely that you will get the same bomb next time. Is there any problem in this situation? Assuming that bucket 1 of layer 1 has a very good positive effect, and it is also found that bucket 0'of layer 2 also has a very good positive effect, then you can’t tell what the positive effect of bucket 0'is The experiment in barrel 0 was brought by the experiment in barrel 0'. Is there any way to solve this problem? Orthogonal. The so-called orthogonality is to regroup the users of each bucket in the previous layer and distribute them evenly to each bucket in the next layer. This is a bit abstract. Looking at the PPT, the 9 users in bucket 0 in Tier 1 are evenly distributed to each bucket in Tier 2 after being broken up again. Similarly, all users in bucket 1 in tier 1 will be evenly distributed to each bucket in tier 2.
The above are the three points of attention for sampling and the solution we adopted. The experimental results of the final A/B test also showed that the data of the part of users who received the content has indeed improved significantly.
In the end, I would like to end with a sentence shared by the boss of our Tencent Watch Point Line at a conference: Most people believe because they see it, and a few people see it because they believe it. Since Facebook created the new content form of information flow in 2006, information flow has been developed for 14 years. As the core engine behind information flow, recommendation system has also entered the deep water area of development. How to provide users with better We have been exploring the content and creating greater value. The delivery system shared with you today is a product of exploration. Because we believe, we can see the other side of victory.
In the end, due to time constraints, the time for communicating with you is very limited. Welcome everyone to communicate with us through our team official account "Tencent VTeam Technical Team" and my personal official account "FEPulse".
thank you all!