Machine Learning Notes 2 - Comparison Test

comparative test

1. Why introduce comparative testing

If we want to compare the learning performance of two learners, we not only need to find two values ​​​​about performance measures, and then quantitatively compare them. Because, in machine learning, the comparison of learner performance involves many aspects, and these aspects sometimes have contradictory relationships.

The main factors we need to consider to evaluate learning performance are the following:

• Generalization ability , that is, the ability of a machine learning algorithm to adapt to new samples. For example, during the college entrance examination, most of us will take the "May 3" test and answer a lot of questions. However, during the college entrance examination, some students can do well in the exam, but some do not. The reason is that the questions in the college entrance examination cannot be the original questions we usually train on. No one has done it. Different students have different generalization abilities, so there is a gap in their ability to draw inferences from one example. In the end, everyone's results are very different. . Putting the ability of high school students to draw inferences from one example to machine learning is the generalization ability of the learner.
• The selection of test sets , different sizes, and test sets of different samples may lead to different training results.
• The randomness of the algorithm , because machine learning is a black box, we cannot know the specific running process. It is possible that the same test set, run with the same parameters on the same learner, will produce different results. result;

Therefore, we need to specifically study the comparative testing problem of machine learning.

2. Several comparative testing methods

1. Hypothesis testing - binomial test

First introduce two concepts:

• The first is called the generalization error rate $e$ refers to the probability that the learner will make an error in classifying a sample under normal circumstances. In actual situations, we cannot know its exact value;
• The second is called the test error rate $e^{\prime }$ , that is, the learner has exactly$e^{\prime }m$samples have been misclassified, and we can generally only obtain this value;

The difference between the two is that the generalization error rate is a theoretical value that cannot be obtained, while the test error rate is a value that we can measure. The method of statistical hypothesis testing is to use $e^{\text{'}}$ Estimatedeethe value of$e$ ;

A generalization error rate $e^{\prime }$ learner to test a size$The\; test\; sample\; set\; of\; m$ can only have two situations: correct classification and incorrect classification, where$e^{\prime }m$are classified incorrectly, and the rest are classified correctly, so this is a typical binomial distribution, then we can get the error rate distribution function of the test:
$$P(e^{\prime };e)=\left(\begin{array}{c}m\\ e^{\prime }m\end{array}\right)e^{e^{\prime }m}(1-e{)}^{m-e^{\prime }m}$$
对$P(e^{\prime };e)$ abouteeThe first derivative of$e$$\delta P(e^{\prime };e)/de=0$，解得$e=e^{\prime }$ time,$P(e^{\prime };e)$ maximum;$|e-e^{\prime }|$增大、$P(e^{\prime };e)$ decreases, just like the binomial distribution;

Then comes the testing process. Since the Xigua book is a bit obscure, I will try to describe the binomial hypothesis test in a simpler way.

Prerequisite : For a distribution function, we know that it is a binomial distribution, but the parameters of this binomial distribution (the binomial distribution has only one parameter, which is the positive probability p, in this example is the generalization error rate ) is unknown.

hypothetical method

• We can use assumptions to artificially specify a "threshold range" for parameter values. In this example, we assume that $e\le e_0$；

• Then we artificially set a confidence level $\alpha$ , this variable represents the standard for obvious errors. For example, the probability of being greater than 6 in this binomial distribution is$\alpha$ , I set the threshold to be$\alpha$ means thatI think if the number of classification errors of the learner is greater than 6, then it is an obvious error. This assumption is not true;

• Conversely, $1-\alpha$ represents the standard that will not cause obvious errors, that is, the standard that can be trusted;

We are now going to use this $1-\alpha$ , to deduce that under this confidence level, we assume$e_0$Is it reasonable?

Determine the plausibility of assumptions

There are two methods to judge rationality. I will mainly introduce the one in the Xigua book;

The general idea is: use the 1 − α 1-\alpha we set$1-The\; value\; of\; \alpha$ is used as a constraint to find the integral area of ​​the distribution curve that is not larger than this value. In the example in the book, the bar 1 statistical chart is used, then multiply the horizontal and vertical coordinates of each bar and then perform a summation operation to find a value not greater than$1-The\; maximum\; value\; of\; \alpha$ , and finally find the ei e_iat this time$e_i$；

The calculation formula is written as follows:
$$\sum _{i=e_{0}m+1}^m\left(\begin{array}{c}m\\ i\end{array}\right)e^i(1-e{)}^{m-i}<1-\alpha$$
means to accumulate the "area of ​​abscissa and ordinate" from the leftmost one of the bar chart, and then add it until it is just no more than$1-up\; to\; \alpha$ .

We take out $i$ , the corresponding abscissa is the acceptable threshold standard, and then take our preset$e_0$Compare with it, if $e_0$If it is less than this threshold standard, it means that the assumption is acceptable and is a reasonable assumption; otherwise, it is unreasonable and needs to be re-assumed.

In fact, at the end of the day, this is a "self-written and self-directed" process. To sum up, first get a rough threshold through the characteristics of the binomial distribution, and then find a more precise point in this threshold. For different learners, we can use this method to manually set parameters and compare the quality of the two learners.

2. Hypothesis testing - t test

Many times, in order to ensure the universality and accuracy of the results, we will do many estimation tests. Suppose we get k error rates $e_1^{\prime },....,e_k^{\prime }$, then we can easily get the average error rate $\mu$ and variance$d^2$的：
$$\begin{gathered} m=\frac{1}{k}\sum _{i=1}^k{e}_i^{\prime }， \\ {d}^2=\frac{1}{k-1}\sum _{i=1}^k(e_i^{\prime }-m{)}^{2.} \end{gathered}$$
Since each estimated detection is independent, this probability distribution function can be written as:T t = k ( μ − e 0 ) δ T_t=\frac{\sqrt{k}(\mu-e_0)}{$$\backslash$$
$$T_t=\frac{\sqrt{k}(m-e_0)}{d}$$
Among them this $e_0$is the generalization error rate, which is a function that satisfies the normal distribution;

Next, we set a confidence level α \alpha just like the binomial test.$\alpha$ , α 2 \frac{\alpha}{2}on both sides of the normal distribution curve$\frac{}{2}$Take an obvious error interval at each location, as shown in the figure:

Then assume a threshold interval $m=e^{\prime }$ , calculate the definite integral under the white image curve (refer to the binomial test for the bar chart calculation method), compare the two hypotheses, if$e^{\prime }$ At the confidence level$If\; \alpha$ is within the range calculated, it is acceptable.

Table of critical values ​​commonly used for bilateral t-test:

The overall process of this part is very similar to the binomial test. Except that the formula and distribution curve are different, the rest of the steps can be copied from the binomial test, so I won’t go into details here.

3. Cross-validation t-test

The above two methods are mainly aimed at measuring the performance of a single learner. For multiple learners, the k-fold cross-validation t test can be used to complete it.

For example, the error rates of two learners A and B after k tests are $e_1^A,...,e_k^A$，$e_1^B,...,e_k^B$, in which $e_i^A,e_i^B$is the i-th fold result obtained after k-fold training. Find the difference between each set of training and test results: $D_i=e_i^A-e_i^B$；

Assume that each test is independent of each other, so we can put $D_1...D_k$Treated as an independent sampling process of k times of performance differences, the mean μ \mu of this set of data is calculated$\mu$ and variance$d^2$，因此我们有：
$$\alpha isacceptables.t.T_t=abs(\frac{\sqrt{k}}{d})<t_{\frac{}{2}}$$
The idea of ​​​​this process is also to set a confidence level $\alpha$ , and then determine whether the threshold is within the acceptable range of confidence.

In fact, our training\test sets can easily overlap, so each test is not necessarily independent of each other .

We can roughly speculate that if our training and test sets overlap, then the k-fold test results we get will have a certain degree of correlation and similarity. If we still assume that these test results are independent of each other, that is to say If we regard a bunch of results that were originally "glued together" as "not glued together" , it means that the acceptable range of our hypothesis is actually too large , and we will overestimate the probability of the hypothesis being true. .

So we can use "5×2 cross-validation" to alleviate this problem. The so-called 5×2 means to do 5 times of 2-fold cross-validation. The data must be disrupted before each 2-fold cross-validation to ensure that the divisions are not repeated. The basic idea is as follows:

• For two learners A and B, each 2-fold cross-validation will produce two sets of test error rates. We calculate the difference between them and get the difference Δ i 1 \Delta_i^1 on the first fold $D_i^1$and the difference on the second fold $D_i^2$；

• Average the two results of the first 2-fold verification $m^{\prime }$ , make a variance δ ′ 2 \delta'^{2}for each 2-fold experimental result$d^{\prime 2}$ , the average value adopts the average value of the previous first fold$m^{\prime }$；

• Its probability distribution function is:
  $$T_t^{\prime }=\frac{m^{\prime }}{\sqrt{0.2{\sum }_{i=1}^5{d}_i^2}}$$

Why does this alleviate the independence problem between test results? It's actually very simple, because our mean is just a set of error rates for one test result. All subsequent test results use the previous set of error rates when calculating the variance, so the size of the variance is only the same as the original data and the first The error rate of each test is related. But I think this method will be more convincing if the results of the first test are more realistic. (My personal thoughts are welcome to correct me).

4. McNemar test

The main idea of ​​this testing method is to list a contingency table for the classification results of the two two-category learners, and then derive the chi-square distribution function of the performance difference between the two learners, and then conduct a hypothesis test. Next, this method will be roughly described. A step of:

• First, list the contingency table for the binary classification results of the two learners A and B:
  

• After that $|e_{01}-e_{10}|$ , this variable obeys a normal distribution; this test method considers the variable
  $$T_{x^2}=\frac{(|e_{01}-e_{10}|-1{)}^2}{e_{01}+e_{10}}$$

• This is a chi-square distribution with a degree of freedom of 1, and the rest of the hypothesis testing steps are the same as above: make a hypothesis to set the degree of reliability, deduce the acceptable range, test whether the hypothesis threshold is within the acceptable range, and draw a conclusion.

5. Friedman test and Nemenyi follow-up test

Cross-validation t-test and McNemar test are only suitable for pairwise comparisons. When comparing learners of multiple algorithms, it is obviously too troublesome to compare algorithms pairwise, so a method is introduced that can compare two algorithms on the same set of data. A test method for comparing multiple algorithms.

The basic idea of ​​the F-test method is to sort the performance of the learner according to the test results on the same set of data sets, and assign ordinal values ​​1, 2, 3..., and equally divide the ordinal values ​​if they are the same, as shown in the figure below:

The better the performance, the smaller the value assigned, and then calculate the average order value r r_i of each algorithm on each data set$r_i$；

It should be noted here that the data in the table is just a special case. This average ordinal value does not completely mean that the algorithm with a small ordinal value must be better , because for different data sets, algorithm A may not be as good as algorithm B, and the two kinds of learning The algorithm may also have similar results on the same data set, so we need to use an estimation threshold to determine which algorithm is better.

Suppose k algorithms are compared on N data sets, $r_i$The mean and variance of are $(k+1)/2$ and$(k^2-1)/12N$ , the distribution function is obtained (the derivation process is omitted):
$$\begin{gathered} T_{X^2}=\frac{12N}{k(k+1)}(\sum _{i=1}^k{r}_i^2-\frac{k(k+1{)}^2}{4}) \\ {T}_F=\frac{(N-1)T_{X^2}}{N(k-1)-T_{X^2}} \end{gathered}$$
$T_{X^2}$is the original Friedman test distribution function, $T_F$is the F test distribution function; in this algorithm, Xiguashu clearly provides a critical value table:

Acceptability test is still the same as above.

If the hypothesis that all algorithms have the same performance is rejected, then subsequent testing is required to make a further comparison of the algorithms. Because in the previous steps we only understood that not all of the k algorithms have the same performance, now we need to "find" the learners with different performance and compare which one is better .

The following introduces the specific process of the Nemenyi test algorithm:

The basic principle is to compare the difference between the average order values ​​of the two algorithms with the CD value. If the mean difference exceeds the CD value, it means that there is a difference in the performance of the two learners.

The CD value calculation formula is as follows:
$$CD=q_{}\sqrt{\frac{k(k+1)}{6N}}$$
where $q_{}$The value table is as follows:

If the difference between two ordinal values ​​is greater than the CD value, then the algorithm with a smaller average ordinal value is better than the algorithm with a large average ordinal value to a certain extent.

So looking back, the average ordinal value is only a reference aspect. We also need to use the Friedman test and the Nemenyi follow-up test before we can more rigorously use the average ordinal value to measure the quality of the algorithm.

Next section: Machine Learning Notes 3 - Linear Regression