Preface
Only a bald head can become stronger.
The text has been included in my GitHub repository, welcome to Star: https://github.com/ZhongFuCheng3y/3y
If the students who have read my article before know that I am engaged in a push system at work, I wrote an article before to show you what Push message push is , and it also mentioned that we may be able to do ABTest , and finally improve the click of push message rate.
So what is ABTest? This article takes you to get started .
1. Introduction of ABTest
For example, I wrote an article about ABTest. I hope that the reading volume of this article can reach 2500, but I didn't think about what the title should be. The title of a tweet can greatly affect the amount of reading, so I thought of several titles:
AbTest knowledge I learned recently
Getting started with AbTest
And I don’t know which title will perform better, so I tried this:
The title of "Recent AbTest Knowledge I Learned" was pushed to 10% of users
The title "Introduction to AbTest" was pushed to 10% of users
After a period of time, I look at the effect, which title has the higher reading volume, and I will push the title with high effect to the remaining 80% of users
ABTest process
It should be noted that when pushing articles, except for different titles, other factors need to be the same (cannot be disturbed by other factors), so that the data can be convincing.
1.1 Why do ABTest?
The reason for ABTest is actually very simple. We have various ideas when we do business, such as:
"I think adding emoji expressions to the copy will definitely have a high click-through rate for this push message"
"I think if this button/image is replaced with another color, the conversion rate will definitely increase"
"I think the homepage should be designed like this, and the picture wall should be like this.."
…..
However, and not all of the ideas are correct , it is probably because you think to get rid of home styles, users do not like, it affected GMVso on and so on ....
A good product is iterative out, and we probably do not know this iteration ultimately good or bad (at least we think is good for users iterations, is helpful, the transformation of the company is good) , But our users may not buy it.
Therefore, in order to reduce the cost of trial and error , we do ABTest. After a function is made, we only put a small amount of traffic to see the effect. If the effect is worse than the original function, it is very likely that our idea has not met our expectations. If the effect of a small flow is better than expected, then gradually increase the flow until the full flow is reached.
2. How to do ABTest?
From the above case, in fact, we probably know that the main thing ABTest does is a triage .
Assign 10% of traffic to user group A
Assign 10% of traffic to user group A
Diversion
What we need to ensure is: a user requests to come in again, the user sees the same result
For example, I visited Java3y, and his profile is: "A Java technology official account that insists on originality". And an hour later, I visited him again, his profile was: "A technical account full of dry goods". And an hour later, I visited him again, his profile is: "A Java technology official account that insists on originality".
Introduction
This is unreasonable. It should be that the user sees the same content within a period of time, otherwise it will bring a sense of confusion to the user.
OK, so generally you can do this:
The user ID (device ID/CookieId/userId/openId) is hashed, and the result of the hash is the same each time.
Take a bit of the user ID directly
Now it seems that ABTest seems to be a diversion thing, but it takes a noble name called ABTest.
2.1 More about ABTest
Suppose I made an ABTest on the UI level, which occupies 80% of the traffic of the whole site. What should I do if I still want to make ABTest of search results? Can only use the remaining 20%? Then my traffic is not enough (I may do various experiments). Can the ABTest on the UI level and the ABTest on the search results be performed at the same time?
The answer is yes. Because the UI level and the business relevance of search results (algorithm optimization) are very low . If you want to " do ABTest at the UI level and search results at the same time for the same traffic ", you must ensure that " ABTest done at the UI level cannot affect the ABTest of search results. "
The most widely used in the industry is the overlapping layered bucket method
The traffic between layers does not interfere with each other, which is the orthogonality mentioned in many articles (the traffic will be broken up again at each layer )
Source: https://www.infoq.cn/article/BuP18dsaPyAg-hflDxPf
We can do it like this: through Hash(userId, LayerId)% 1000 similar method to achieve
No matter how many experiments there are in each layer, the impact on other layers is uniform
My understanding:
In order to realize the UI/algorithm/advertising business has nothing to do with the ABTest test, so it is divided into layers. When the flow passes through each layer, the flow needs to be broken up again (orthogonal)-after each layer experiment, it will not affect the experiment of the next layer
If the business relationship is strong, it should be placed on the same layer, and multiple experiments on the same layer are mutually exclusive (for example, a button color is changed to green as an experiment, and the style of a button is changed to thumbs as an experiment. The traffic of these two experiments Are mutually exclusive (otherwise how do you know if the user clicks because of your button color or style)
Schematic diagram
At last
A complete ABTest system is not only for diversion, but also provides users (our programmers) with a convenient and configurable backend system, completes the experiment and provides data report display, etc.~
WeChat official account does not support external links, reply " AB " in the background to get more ABTest information
Two years of painstaking articles : "Interview Questions", "Basics" and "Advanced" are all here!
Long press to scan the code to follow to get
Watching and sharing are very important to me!