| Lead
In recent years, with the rapid development of the Internet, Growth-Hacking has been a very popular concept. Growth-Hacking aim is to use smaller, more flexible cost to dig through data-driven mystery product growth. While AARRR this model, constantly playing a good flow of the loop, we need to find the correlation between product features, operating strategy and transformation from data analysis, thinking a causal relationship between them.
One accurate measurement is worth more than a thousand expert opinions
– Admiral Grace Hopper
How to measure thinking, innovative ideas correctness? Data is the best measure, which requires that we therefore need to use some tools. The AB test is a rapid trial and error, user impact as small as possible, the data of scientific decision-making tool, it is one of the most basic Growth-Hacking is the most important tool.
Since 2000, Google engineers will ABTest applications since the Internet products, A / B testing is becoming increasingly popular at home and abroad, it has become an important manifestation of the Internet data-driven products to drive product design iterations through A / B test optimization.
What is ABTest
A / B test data-driven guide and to be flexible traffic segmentation, such that different versions of the same product simultaneously online behavior data generated by different versions by recording and analyzing a user to obtain the effect of contrast, the maximum degree of Results the scientific and accuracy to help people make scientific decision-making product.
The main part of ABTest
Below is a generic architecture design:
The whole structure contains the following sections:
- AB test management platform : experimental operation portal to create, modify, closed experiments, and provides report viewing.
- Configuration database : experimental configuration data, is not limited to ordinary relational database, there may be cached database.
- Diversion service : The experimental configuration data, shunt experiment specific logic, this is generally integrated in each service platform or service server.
- The SDK : logic to provide a common parsing shunt, typically integrated in a client front end.
- Data collection : Split results log, real-time collection of user behavior logs.
- Data analysis : real-time and off-line data analysis algorithms to make scientific decisions by certain data analysis.
The statistical principles ABTest
Test principle from the A / B testing of view, it is statistically hypothesis test (significance test) a form: hypothesis testing of the test parameters is to propose a hypothesis of the general parameters, then the sample data is determined by using assuming that the process is established.
Use logic, based on small statistical probability ideological contradiction:
- Small probability thinking refers to a small probability event (significance level of p <0.05) basically does not happen in a trial.
- Reductio ad absurdum refers to the first hypothesis, then the appropriate statistical methods to determine how likely hypothesis; such as small possibility, then that assumption does not hold.
Specific to the comparison test, the test is generally assumed that version parameters (mean optimization metrics) is equal to the control version of the population parameters, and then use these two versions of the sample data to determine whether this assumption holds.
Testing hypotheses based on the concept
- The null hypothesis: also known as the null hypothesis, H0, we usually assume that the statistics are statistics between two groups in the same experiment, that is equal to the average mean of the experimental group control group.
- Alternative Hypothesis: also for the alternative hypothesis, i.e. reject the null hypothesis; is not equal to the experimental group mean control group mean.
- Two-sided test and one-sided test: if the alternative hypothesis does not determine the direction that is "≠", compared with two-sided hypothesis. If a specific direction, comprising a ">" or "<" one-sided test, compared.
- Test statistics: statistics when used to test the hypothesis called the test statistic, such as the mean of the sample set.
- Accepted Domain: accept the null hypothesis that the sample (X1, X2, ..., Xn) area is located.
- Negative fields: null hypothesis that the region composed of a negative sample.
- Simple and complex assumptions hypothesis: whether the null hypothesis or alternative hypothesis contains only one argument was simple hypothesis otherwise complex assumptions.
Two types of errors
- Type I error (Error discarded true): null hypothesis is true when the null hypothesis is rejected; Type I error probability is referred to as α (alpha).
- Type II error (false alarms take): null hypothesis is not rejected the null hypothesis is false. Type II error probability is denoted by β (Beta).
| Truth \ actual decision-making | Accepting H0 | Reject H0 |
|---|---|---|
| H0 is true | The right decision | Type I error |
| H1 is true | Type II error | The right decision |
- Power function: Let R represents a rejection region of the test,
Statistical significance level and effect
- Significance levels : level of significance is true sometimes refers to the probability or risk being rejected, the probability of an error type occurs in original hypothesis is α. Typically AB test, we set the significance level of 0.05, is obtained when the p-value i.e. p <= 0.05, the null hypothesis is rejected; p> 0.05, then the null hypothesis can not be rejected.
- Statistical power : statistical power, it simply is the possibility of truth can be found. Like insulin can lower blood sugar This thing is real, but humans find its probability is how much? If the statistical power is 0.8, that human beings have 80% probability can find it. It is a mathematical formula can be used to summarize the definition, the statistical power = 1-β.
p-value calculation
AB test calculated p-value calculations and interval estimation are corresponding, which we will use this two-sample Z-test, briefly explain the formula p-value of:
- Mean categories of indicators :
- UV CTR / rate conversion ratio index categories :
According to find the value of p Z value, in fact, find the standard normal distribution points, find (z, ∞) of N (0,1) * 2 points.
mPaaS ABTest
Internal ants gold dress also has a strong culture experiments, ABTest experimental platform has accumulated tens of thousands of test runs: from Alipay client style experiments, interactive experiments to back-end algorithms, experimental strategies have a rich accumulation of cases, in the process ABTest platform It has been the depth of temper and the ability to accumulate more and more easy to use, mature and stable, credible authority.
Currently, ABTest platform has completed the transformation of the product, and settled in to mPaaS product system, provide strong support for the intelligent building ability mPaaS.
Experimental model
It is necessary to introduce ABTest architecture before further discuss the experimental model ABTest:
Segregation of traffic and reuse
If the same number of users simultaneously participate in multiple experiments, between the experimental scene if relevant, the results between the two experiments may affect each other, resulting in incorrect experimental results. So we need to make the same user in the same period of time can only participate in an experiment, in order to avoid the effects of multiple experiments cause, called the segregation of traffic.
With the increase in online experiment, traffic isolation can not avoid running out of traffic and traffic problems of hunger, and this traffic is multiplexed forth requirements.
Google's architecture-level domain
In Google 2010 released in "Overlapping Experiment Infrastructure: More, Better, Faster Experimentation" this article describes Google's experimental architecture overlap aimed to provide a platform for experiments Better & More Faster in.
About Google proposed overlap experimental architecture at its core level domain model:
By the graph we can understand the basic structure of a domain model layer, this kind of structure is not only compatible with single-layer test and multivariate experiment, while more flexible, can be divided into various combinations of system parameters.
Because of the complex relationships between the level domain, although this design adds flexibility while allowing traffic but also greatly increases the cost side, mainly based on the actual business single experiment, when the whole link, each branch are independent , and a subset of the division of each branch is relatively clear, so the demand for domains is not very strong.
ABTest domain model
mPaaS ABTest domain model:
ABTest on mPaaS through the "laboratory" to achieve a layered, that is, each "lab" has an independent 100% of the traffic. Since a link real business and not a lot of parameters and parameter between independence is relatively clear, so ABTest weakening the domain concept.
- laboratory
100% of the inlet flow. Support setting parameter default values reveal all the details. Business independent of each business should have an independent laboratory.
- experiment
You can create multiple experiments at each laboratory, traffic between experiments are mutually exclusive, to achieve the purpose of traffic isolation. At the same time the experimental level support circle of people targeting criteria, supports directional test.
- Experimental version
You can create multiple versions of the next test experiment, the binding of the experimental values of experimental parameters on the experimental version. Each version of the flow distribution through the mapping between the bucket and 10000 are allocated.
ABTest architecture
Architecturally speaking ABTest into the access layer, the core layer and the underlying capabilities dependent layers, the following highlights the capacity of the access and core layers.
Access layer
Access layer to solve the problem of how to access ABTest business system components, ABTest Internet business is usually divided into two categories: client and server test experiments, the access layer of the two types of experiments have provided support.
Client experiment
The client test is usually carried out experiments on the client's style, interactive processes that help product development teams and better iteration and optimization.
By experimental setup configuration service to clients (Remote Config) reaches contact client, the client service by way of taking a local variable from the local cache, get the result of the experimental variables and the corresponding value of the shunt, so rendering the corresponding difference, to achieve provide different services to different users, head to experience, and dynamic configuration (Remote Config) service logs split log, return the server.
H5 container, small application framework integrated Remote Config, exposed to H5 service by Hybrid API, applet, so that the client Native, H5, three small development framework program are equipped with the ability ABTest.
Server test
Server-side experiments are usually service strategy, algorithm experiment, is the foundation of AI ability iteration.
Performance, server-side problem is the difficulty of access to business systems access ABTest more concerned about. ABTest pulled out of service independently of the shunt the SDK, the experimental setup information stored in the memory, after integration into the host system can provide local service diversion, to avoid the network overhead. Meanwhile SDK provides a simple-to-use interface, reducing the difficulty of use.
The figure is split between the SDK and the relationship ABTest Console, the host system:
SDK integrated shunt certain development costs, need to access business systems. In various preset internet mPaaS inlet flow diverter assembly SDK provides a diversion services, access to simplify the work ABTest:
MGS gateway service will shunt parameters in the request context spread through business systems, business systems can be used directly; publishing service of H5 MDS offline package management platform directly to a different version of the H5 applications do ABTest; intelligent delivery MCDP can support different advertising campaigns do ABTest.
Core competencies
MPaaS ABTest on core competencies, and it has reached a common ABTest system should have a standard, divided into two parts:
The ability to experiment
Including the ability to experiment experiment management, definition of indicators, directional circle of people, diversion services, push all-gray. We focus on "life of the experiment & Science flow separation" focus on expansion:
- An experiment lifecycle including " create an experiment, function and link verification, officially running, gradually heavy volume, pushing the full amount of the whole " five stages. In the experiment state conversion process, distribute traffic is scientific is very important.
- In a laboratory, each user is bound by Hash algorithm to the numbers 0 to 9999 barrels, while the experimental version number of the saved map of the barrel
- ABTest by traffic into an idle buckets, recycling bins, bucket using three states, preferably used in the flow distribution during the idle tub , secondly recycling bins . After recycling bins have been distributed, the flow of the entire lab has been used up, the need to push full with other experiments in the laboratory or off the assembly line, using buckets after being recycled to new experiments. ABTest system transfer process of scientific experiments in the distribution life cycle, recycle flow (ie change the state of the bucket) to ensure the scientific nature of flow distribution, reduce the disturbance to the user.
Skills of analyze
Including real-time analysis capability shunt PV / UV statistics, T + 1 Experimental significance reports and multidimensional analysis and comparative analysis.
Experimental data split into client and server shunt Buried ABTest SDK shunt log shows two were collected by HDFS gateway and log flume. Experimental results statistics SDK comes with custom events Buried back to the HDFS through the gateway and log flume through mPaaS client.
- Real-time computing link
Kepler data import by Kafka, kepler mission and business transformation shunt log log double join, and experimental PUV statistics, final results will be dumped to HBase, ABTest Console to show results.
- Off-line calculation link
Data introduced through flume HDFS, synchronized by an offline computing platform, computing offline data Hbase reflux and ABTest metadata index and results, ABTest console to show the results. Offline link for calculating self-serving significant reports, and other custom metrics has retained a large amount of calculation of the scene.
- Instant analysis link
Offline part of the data will also link back to the pretreatment system LDAP Explorer, ABTest Console Explorer made more flexible by using multi-dimensional analysis and the funnel analysis
mPaas index calculation system
Take for the client user behavior mPaaS custom events Buried, users only need to automatically generate test statistics T + 1 through performance data associated with a particular custom event can be. Each individual user to experiment with independence, we believe that there is behavior between the two users are independent, and each user behavior during the experiment is relevant, by statistical analysis of user behavior during the experiment all .
For besides sum and count categories of indicators we will carry out interval estimation (calculation between index statistical confidence interval, protocols contrast to the absolute difference confidence interval and a relatively poor confidence intervals), and calculate the p-value based test hypothesis given significant statistical conclusions .
- Global indicators
During the experiment enter the number of the various options (cumulative UV) and frequency (cumulative PV) as an experimental shunt index based; another part of the global indicators for the retention class indicators to the user first enters the start of the experiment, the next day is referred to as active The next day retention, and so on, we can calculate 2,3, ..., 7, retention and so on.
- Simple type indicators
Simple type indicators custom events consist of a single, after a user to configure a custom event, automatically generated test indicators corresponding to: include, UV conversion Total PV (event-triggered times), UV Total (event triggered total number of users) ( experiments user triggers users accounting for the custom event), mean (total number of subscribers to trigger / into the experimental protocol).
- Composite index
Composite index is related to the statistical effect of multiple custom events calculated composition, including poor pay and in addition, we also support the operation in the four composite indicators on the basis of the set operation. That which we order two custom events E1, E2, for example:
And (E1 + E2) : indicates that the user just happened E1 or E2 that is considered the user triggers (E1 + E2) event, then the event is the total number of trigger trigger trigger the total number of E1 + E2 the total number of the remaining relevant indicators calculated on It can be seen as a simple type index for processing.
Post (E1 * E2) : indicates that the user not only triggered the E1, E2 also triggered, based on that which we can look at a single session, so the total number of trigger events is triggered while the total number of sessions E1 and E2 can also defaults to 1 (direct current mPaaS believe 1, to calculate where the intersection of our main view of converting UV), other index calculation can be seen as a simple index for processing.
Difference (E1-E2) : indicates that the user does not trigger triggered E1 E2, the total number of triggers event is the total number of users to meet the conditions of the trigger E1, the rest of the relevant index calculation can be seen as a simple type indicator to deal with.
In addition to (E1 / E2) : This type of event that we become transformed, the simplest example is the E1 event indicates a click of a button, E2 event indicates that the exposure of a button, the conversion rate is pv sum (E1) / sum (E2 ) i.e., pv-ctr, conversion of UV same mean diameter will clear the like. Also exposure to click, for example, when the actual pv-ctr variance We discovered that in fact there is a correlation, then when calculating actual variance we use the Taylor expansion between a user and multi-exposure E1, E2 co-related coefficient variance formula has been optimized:
Suppose x1, x2, ... xn for each user clicks, y1, y2, ..., yn impressions for each user,
Above, it is moving to resolve specific technical architecture and end ABTest summarized in Alipay how landing practice. If you are interested ABTest, we can further attention mPaaS subsequent articles and product updates iteration.
Past Reading
"Opening | ants gold dress mPaaS server core component system overview"
"MPaaS server core components: mobile analytics service MAS architecture analysis"
"Ants gold dress face one hundred million concurrent scenes Component Architecture Design"
"Automated log collection and analysis of evolution in the App Alipay"
Follow us public number, to obtain first-hand mPaaS technology practice dry
Group nails: nail Search by group number "23124039"
We hope you can join ~