Lin Xuantian's Notes on Machine Learning Techniques (4)

I suddenly realized that I remembered a little bit more details? It feels like progress has slowed down.
Lin Xuantian's Machine Learning Techniques Notes (1)
Lin Xuantian's Machine Learning Techniques' Notes (2)
Lin Xuantian's Machine Learning Techniques' Notes (3)

P14 4.1
How to reduce the overfitting of Gaussian SVM

1. A bit φ (conversion) is too powerful
2. You may have noticed that I have always insisted on separating the data perfectly, which is not necessarily the best classification method, and will be affected by the inevitable noise

Recall how to avoid noise in the past:
pocket: just find a line with the smallest error.
Therefore, we can tolerate some errors like a pocket, and we can combine them, and use C to weigh errors and correctness:

thus we get: soft-margin SVM
$[y_n̸=sign(w^T{z}_n+b)]=$When it is$0$$y_n(w^T{z}_n+b)>=1$，$[y_n̸=sign(w^T{z}_n+b)]=When\; 1$ , an error occurs at this time, and then the error will be recorded. C indicates the weight of the error in the whole, and at this time it is required that$y_n(w^T{z}_n+b)>=-\infty$ .
But in this way, the blue area is not linear, and then the dual and kernel are not valid. And if the number of errors is recorded, small errors are treated as the same as large errors, and the effect will not be good. Therefore, it is necessary to determine alinearconstant$g_n$, it becomes a record of how many mistakes were made, not the number of mistakes, so it is a QP problem again.

$g_n$Indicates the degree of error at each point, which is the violation distance in the red box. A large C means low tolerance, and a small one is the opposite. Then because of the addition of $g_n$, the number of variables is d ^~ +1+N, because adding $g_n>=0,so$ there are 2N constraints (constraint):

---

P15 4.2
Next, derive the dual problem, and then use the kernel to solve it.
Similarly, set α and β:

take the gradient of L to ζ as 0, use α to represent β, and get a new formula after substitution: the

obtained new formula is actually the same as the previous hard-margin, except that the condition A little change, here is 0 <= α <= C:


this QP problem, is N variables because there are $a_n$, there are 2N+1 constraints: because each $a_n$There is a lower limit 0 and an upper limit C, so there are 2N, plus 1 $\sum$ , there are 2N+1 constraints:

---

P16 4.3
Now we can use QP to calculate α, but the value of b is unknown, because it was calculated using the fourth condition of KKT before, but now because it is soft instead of hard, there is no such condition: compared with

hard , if you want to solve it, you have to solve ζ, and if you want to solve ζ, you have to know b, which is a bit like "If you can't beat the boss, you need to unlock the skills, but to unlock the skills, you need to fight the boss": so define the free support vector (point α

s $点{\_}_{s}Rangeis0<a_s<C$ ), according to the second formula, let$g_s=0$ , so there is a b, of course, this b is not fixed, as long as the conditions are met. And generally, there will be at least one set of SV such that$a_s<C$ (red stone).

Although there is now soft protection, the value of C cannot be randomly selected, otherwise it may overfit:

Now, the points on the graph can be divided into three types:
$a_n$The value is 0~C
①: Non-SV ( $a_n=0$ ): Here, according to the second formula, ζ = 0, indicating that there is no mistake at this point, but$a_n=0$ means that these points are not SV, so most of these points are outside the fat boundary (very few are on the boundary)
②: free SV ($0<a_n<C$ ): Here, according to the second formula, ζ = 0, indicating that there is no error at this point, but these points are SV, so they are just on the boundary. Since ζ = 0, b can be calculated according to the first formula.
③: bounded SV($a_n=C$ ): These points are within the boundary (very few boundaries), where ζ represents the degree of violation of these points, that is, the distance to the boundary.

---

P17 4.4
You can use cross-validation (CV) to verify whether the selected parameters are good or not, and select the set of parameters with the smallest cv:

in the leave one out Cross Validation of SVM ( I don’t remember what it is, click here , roughly just take the data set is N-1, the test set is k=1), and then here is a proof that Eloocv <= SV (number)/N: because non-SV's err=0, that is, points outside the boundary do not contribute to SVM's err , and e _SV <= 1, so Eloocv <= SV (number)/N

and then SV / N can be used to simply filter out the high ratio, because too many SVs may mean that the model is not very good. After simple exclusion, CV can be further done on other models.

Summarize: