This series of articles contains some exploration and practical experience in recalling, sorting, and cold-starting in the past year. This article is the sixth article in this topic.
The first guide: cold start system optimization and content potential estimation practice
The second guide: GNN recall practice in light application content recommendation
The third guide: summary of ODL practice based on feature full embedding
The fourth guide: Gradient Normalization optimization practice in multi-task learning
The fifth guide: the application practice of generative rearrangement in content recommendation
scene introduction
The light application of every flat and every house is a content-based shopping guide for home furnishing products in Taobao. The content is mainly based on scene matching, and multiple commodity anchors are mounted in the content. Taobao users can enter the light application interface of each flat and each house in the following 3 ways: 1) search for keywords of each flat and each house; 2) after paying attention to the light application of each flat and each house, actively visit my channel on the second floor of Taobao; 3 ) Click on the Taobao homepage to guess the content card of every flat and every house you like. After the user enters the channel page of the light application of every flat and every house, browse and click on the content card to enter the content details page, click the product anchor point of the details page to enter the product details page, and then perform behaviors such as favorites and purchases.
At the beginning of 2021, the content details page of the light application of each flat and every house has been fully switched to the endless stream form. Compared with the original immersive single image and text consumption, the endless stream adopts the undamped pull-down form, and the user's pull-down browsing experience is smoother, thereby improving the content of the content. The exposure and consumption PV of the detail page;
In order to better optimize the browsing depth and click efficiency of the endless stream scene, and bring more consumption PV to the content details page of the light application, the algorithm students of each flat and each house have optimized each recommended module, including enriching recall diversity, introducing rough Layout model, drop-down model development and optimization, and adjustment of content relevance display strategies, etc.
Recall optimization
▐ Relevance & Diversity
The predecessor of the endless stream of every flat and every house-the associated recommendation scenario pays more attention to the correlation between the recommended content and the drainage content, and the overall recall link is relatively simple. After upgrading to the endless stream form, in order to improve the user's pull-down depth and click efficiency, it has higher requirements on the level of diversity and user personalization. Therefore, in the recall link, we have added a number of recall links from the three dimensions of drainage content relevance, recall diversity and popular bottom line:
After enriching the diversity of recalled content from multiple dimensions, observing the AB indicators online for 7 days, the cumulative index of the recall link of content relevance increased: uctr +10.99%, pctr +20.10%, pull-down depth +1.86%, per capita click +22.62 %, average exposure +1.72%; Diversity optimization and recall link cumulative index increase: uctr +6.23%, pctr +13.25%, pull-down depth +2.41%, per capita click +16.17%, average exposure +2.38%; overall In the above, the recall optimization of different dimensions has brought a positive improvement to the scene click efficiency and browsing depth. Among them, the recall based on the relevance of the drainage content has brought a great improvement to the scene click rate indicator. After optimizing the recall diversity, the user pulls down the depth. increase greatly;
▐ Rough row
With the enrichment of recalled links, the content candidate set after the recall of each link is getting larger and larger, and the expansion of the candidate set will increase the pressure of RTP fine-ranking scoring, which makes the fine-ranking scoring RT increase rapidly. The quality of the recalled content determines the upper limit of the fine sorting result. In order to take into account both efficiency and precision, the content with better recall quality is pushed to the fine sorting score. We introduce a rough sorting model in the endless stream scene to perform preliminary scoring and screening of the recalled content. After the launch, the click efficiency of the scene has improved: uctr +2.71%, pctr +5.86%, pull-down depth +0.13%, per capita click +2.85%, per exposure +0.40%.
Sort optimization
▐ Multi-objective learning model
In the light application scenario of every flat and every house, after the user enters the channel page, browse and click the content card to enter the content details/endless flow page, click the product anchor point of the detail page to enter the product details page, and then conduct collections and purchases, etc., business The optimization goals involved in the model include one-hop clicks, two-hop clicks, and ordering after entering the business details page. Therefore, we use the user's multi-jump behavior samples to train the MMOE multi-target model, which greatly improves the multi-jump clicks of each light application in every room.
After the scene two-hop page details page is upgraded to the endless stream scene, the traffic of the endless stream scene in the early stage of the business is small, and the scene mainly uses the MMOE multi-objective model trained by the samples of the main channel of the light application as the scoring model. The advantages of the model trained directly using the main channel samples are: the model has the user's global behavior preference information in each flat and each house, and the model models the multi-hop click information, which contains endless stream click behavior samples (two To a certain extent, it takes into account the user behavior preferences of channel global and endless streaming scenarios, and has a considerable effect on click efficiency; however, this model mainly optimizes the scene click efficiency. In endless streaming scenarios, we also pay attention to the drop-down depth. Optimized, so a drop-down model based on endless streaming scenarios is built;
▐ Drop-down model
In the endless flow scenario, in addition to the optimization of click efficiency, another important goal is to optimize the pull-down depth, thereby increasing the exposure and consumption PV of the detail page; however, the optimization goal of pull-down depth is different from the optimization of common click efficiency indicators. On the one hand, the user's pull-down depth is the overall number of exposures that the user continues to pull down after entering the endless stream details page, which is an indicator of multi-card accumulation; on the other hand, whether the user continues to pull down after scrolling to a certain depth in the endless stream, Or leave directly, not only affected by the current exposure card, but also related to the previous exposure sequence. If the content in the recommendation stream is too different from the content in the stream, or the fatigue level is too high, it may cause users to skip. So in order to optimize the index of browsing depth, we start from the context and relevance to model the user's pull-down behavior.
Model Selection & Sample Construction
In model selection, considering that there are not many samples in the current endless flow scene, it is impossible to train too complex models. After evaluation, W&D is used for training [1]. For sample construction, different from the click-through rate model by judging whether the user clicks to construct the sample, the endless stream pull-down model mainly constructs positive and negative samples by whether the user continues to pull down, and continuously optimizes the samples according to the characteristics of the scene:
v1: Based on whether the user is willing to continue to pull down, the samples with continuous pull-down behavior are regarded as positive samples, and the samples without continuous pull-down are regarded as negative samples;
v2: Based on "users with no pull-down behavior at all, and the user's intention is unclear (interfering with model training)", the user samples without pull-down behavior at all were excluded, and the excluded user behavior samples without pull-down behavior accounted for 24% of the total sample size;
v3: Based on "the user may not be skipped because of the last browsed content", it is believed that in the user's drop-down sequence, the content browsed later may cause the user to skip, and the last 10% of the pulled-down content is regarded as a negative sample;
Model Offline AUC
v1 |
v2 |
v3 |
-- |
+1.97% |
+0.66% |
online AB
ab bucket |
uctr |
pctr |
pull-down depth |
per capita clicks |
average exposure |
v1(base) |
-- |
-- |
-- |
-- |
-- |
v2 |
+5.15% |
+3.52% |
+1.21% |
+4.71% |
+1.34% |
v3 |
+0.88% |
-1.48% |
+1.76% |
+0.21% |
+1.74% |
Conclusion: From the offline AUC, the v2 version has the best effect, which is nearly 0.01 higher than v1, and the v3 version is not as effective as v1;
From the comparison of the complete online AB7-day indicators, the v3 version has a larger drop-down depth (per capita exposure and average exposure), but its click-based indicators have negative returns; overall, the v2 version has the best performance, and there is an improvement in the drop-down depth indicator. , the click index has also been improved; therefore, we built the sample based on the v2 version, and made other optimizations;
Correlation modeling
The user clicks the channel content card to enter the endless stream page. Therefore, it can be considered that the user has a strong interest in the clicked content. We call the content (drainage content) clicked by the user when entering the endless stream as Hero Content. In the endless streaming scenario, we can make full use of the Hero Content information to characterize the user's current interests. In the recall phase, after we introduced Hero Content-related recall channels, the scene effect has been greatly improved. In the sorting stage, we also tried to perform correlation modeling based on Hero Content. The specific ideas are as follows: 1) Add Hero Content and its attribute information to the context feature of the endless flow model, and participate in model training and sorting; 2) Combine Hero Content and its attributes Information and ranking candidates do feature intersection; build the following experimental version:
v4: Only new context features related to Hero Content and its attribute information are added;
v5: Only feature cross between Hero Content and its attribute information and sorting candidate content and attributes;
v6: Added the context features related to Hero Content and its attribute information, and used this information to perform feature crossover with the sorting candidate content attributes;
Model Offline AUC
v2(base) |
v4 |
v5 |
v6 |
-- |
+1.05% |
+3.61% |
+3.86% |
online AB
ab bucket (Compare v2 version) |
uctr |
pctr |
pull-down depth |
per capita clicks |
average exposure |
v4 |
+3.07% |
-0.69% |
+4.00% |
+3.18% |
+0.01% |
v5 |
+0.45% |
-2.07% |
+1.89% |
-0.35% |
+0.47% |
v6 |
+2.22% |
-2.18% |
+3.85% |
+1.45% |
-0.42% |
Conclusion: From the offline AUC, the v6 experimental group has the best effect, which is 3.86% higher than the base v2;
Online complete AB for 7 days, whether it is click efficiency or pull-down depth, the overall online effect of the v4 experimental group is the best, followed by the v6 experimental group. In response to the difference between the online and offline experimental conclusions, we lengthened the AB experiment time. After 14 days, the effects of the v6 experimental group and the v4 experimental group are basically the same: the overall user click efficiency is +2.32%, and the pull-down depth is +3.76%.
Multi-scene sample training
Based on the endless streaming format, which can bring more exposure and consumption PV of the content details page, the details page scene of out-of-channel casting and first guessing drainage is also fully switched to the endless streaming format and the algorithm is optimized. Due to the lower DAU of the endless flow scene of the first guess of the foreign investment, it is impossible to train an independent endless flow model. At the same time, considering the similarity between the two scenes, try to introduce multi-scene endless flow samples for pull-down model training, and add the scene mark feature. In order to distinguish different scene samples; after the sample training of the mixed channel endless stream and the first guess drainage endless stream scene, compared with the original single channel endless stream scene sample training, the offline AUC of the model increased by 4.63%, online AB: the first guess drainage endless stream uctr +2.35%, pctr +2.23%, pull-down depth 0.06%, per capita click +1.37%, per-time exposure +0.14%; no significant improvement in channel endless streaming scene;
Analysis: Introduce endless flow samples of small scenes for multi-scene sample training. Due to the large difference (nearly 100 times) of samples between different scenes, the model has greater benefits in small scenes, but not in large scenes.
relevance strategy
By analyzing the click-through rate at different positions in the endless flow scene, and combining the behavior and mind of the user scene, we have come to two conclusions: 1) as the pull-down becomes deeper, the click-through rate has a certain downward trend; 2) in the same position , the click-through rate of recommending content of the same scene/style will be higher; so we can speculate that when users start to enter the endless streaming scene, their interests are mainly focused on content similar to Hero Content. gradually spread out.
In order to attract more users to browse and click, and bring more consumption PV to the content details page of the light application, we combine the changes of user interest diffusion to optimize the strategy of endless stream drop-down recommendation:
Weighting and determining the content of Hero Content with the same attributes (style, scene, mounted product, author) ensure that the proportion of content related to Hero Content in the top 10 recommended content is more than 30%;
For the content of the second and third pages (position: 11~30), the ranking is mainly based on the user preference model;
From the third page onwards, priority is given to diversity and novelty;
The overall positive income of online AB: uctr +13.06%, pctr +6.01%, pull-down depth +4.70%, per capita click +10.94%, per exposure +3.96%.
Summary and Outlook
Compared with the original single image and text browsing, the endless stream form is a new attempt to display the content details page of every flat and every house. Exposure and consumption PV. This paper summarizes the optimization work and the results achieved by the algorithm in the new content recommendation form after switching the endless flow of the light application details page of each flat and each house. The pull-down model and relevance strategy proposed according to the business form and analysis data It is simple and not complicated, but it has achieved good online results. This also shows that the core of business scenario optimization is not the difficulty of the model, but the way of thinking and the path to solve the problem. The complexity of the model does not necessarily mean Best of all, it is important to understand the business and be able to come up with appropriate algorithmic strategies for different business forms and scenarios to help the business achieve results.
In the subsequent optimization, consider continuing to optimize from the two aspects of drop-down model structure and feature engineering. For example, the sample construction method of the current drop-down model is relatively rough, and in the future, we can consider how to find the "true negative samples" that cause users to skip based on the assumption that "users may not skip because of the last browsed content"; currently After adding the drainage content Hero Conetnt to feature optimization, we have achieved a good improvement. In the future, we can continue to consider how to better integrate the drainage content information into model training; at the same time, the current pull-down model features are relatively simple, and we can combine data mining and analysis in the future. More favorable features will be added to model training, etc. In general, our algorithm optimization on the product form of endless down-flow is just the beginning, and there are still many optimization points and ideas to try in the future. We will continue to explore more optimized forms of endless streaming, improve the pull-down depth and click efficiency of endless streaming scenarios, and bring more consumption PV to the content details page of light apps;
Thanks
I would like to thank my seniors for their careful guidance and teachers for their understanding and support. I would like to thank all the algorithm and data students in every flat and every room, as well as the intelligent scene algorithm students for their exchanges and cooperation.
References
【1】https://dl.acm.org/doi/abs/10.1145/2988450.2988454
team introduction
Big Taobao Technology - Taobao Smart Team
Taobao Smart Team is a team that integrates data and algorithms, serving more than 20 business scenarios in Taobao, Tmall, Juhuasuan, Xianyu, and every flat and every house, providing online retail, content community, 3D Data and algorithm services such as intelligent design and on-device intelligence. Through machine learning, reinforcement learning, data mining, machine vision, NLP, operations research, 3D algorithms, search and recommendation algorithms, we seek business opportunities for millions of merchants, provide intelligent solutions for platform operations, improve user experience for users, and design The division provides automatic collocation and layout, thereby promoting the supply prosperity and user growth of the platform and ecology, and constantly expanding the business boundaries.
This is a rapidly growing learning team. While creating business value, we continue to output academic achievements and publish several academic papers in international conferences and journals such as KDD, ICCV, and Management Science. The team has a strong learning atmosphere, and organizes hundreds of technical sharing exchanges every year to learn and inspire each other. We sincerely invite outstanding talents in relevant directions at home and abroad to join us, grow and contribute their talents here.
If you are interested, please send your resume to [email protected], looking forward to your joining!
✿Extended reading
Author | Solstice
Editor | Orange Jun
