Machine Learning (1) KNN Algorithm

---

foreword

$The\; k-$ nearest neighbor method is a basic classification method. $The\; input\; of\; the\; k-$ nearest neighbor method is the feature vector of the instance, which corresponds to the point of the feature space; the output is the category of the instance. $The\; k-$ nearest neighbor method actually uses the training data set to divide the feature vector space and serves as the "model" for its classification, so it does not have an explicit learning process. $k$ value selection, distance measure and classification decision rule is$The\; k-$ nearest neighbor method has three basic elements.

---

One, $k$ nearest neighbor algorithm

Input: training data set
$$T=\{(x_1,y_1),(x_2,y_2),\cdots ,(x_N,y_N)\}$$
, where$x_i\in h\subseteq {\mathbf{R}}^n$ is the feature vector of the instance,yi ∈ Y = { c 1 , c 2 , ⋯ , c K } y_i\in Y= \{c_1,c_2,\cdots, c_K\}yi​∈Y={ c1​,c2​,⋯,cK​} is the category of the instance,$i=1,2,\cdots ,N.$ _
Output: instancexxthe class yy$to\; which\; x$ belongs$y$。

1. According to the given distance measure, in the training set $T$ find out with$x$ nearest neighbor$k$ points, covering thiskkxx$of\; k$ pointsThe neighborhood of$x$$N_k(x)$;
2. At $N_k\left(x\right)$ decides xxaccording to classification decision rules (such as majority voting)categoryyy$of\; x$$y$:
   $$y=\text{arg}\underset{c_j}{\max }\sum _{x_i\in N_k(x)}I(y_i=c_j),i=1,2,\cdots ,N,j=1,2,\cdots ,Part\; K$$
   ,$I$ is an indicator function, that is, when$y_i=c_j$Time $I$ is 1, no$I$ is 0.

$A\; special\; case\; of\; the\; k-$ nearest neighbor method is$k=1$ , it is called the nearest neighbor algorithm.

2. Three basic elements

1. Distance measure

$The\; feature\; space\; of\; the\; k-$ nearest neighbor model is generally$n-$ dimensional real number vector space${\mathbf{R}}^n$ . The Euclidean distance is used, but other distances are possible. As a more general$L_p$distance. Define two points in space $x_1=(x_1^1,x_1^2,\cdots ,x_1^n{)}^T,x_2=(x_2^1,x_2^2,\cdots ,x_2^n{)}^{}$L p L_pof${}^T$$L_p$The distance is defined as
$$L_p(x_1,x_2)={\left(\sum _{l=1}^n|x_1^l-x_2^l{|}^p\right)}^{\frac{1}{p}}$$
when $p=2$ , called Euclidean distance, when$p=1$ , called the Manhattan distance.

2.$Selection\; of\; k$ value

  $k$ is worthy of choice will$The\; results\; of\; the\; k-$ nearest neighbor method have a significant impact.
  If a smaller$The\; k$ value is equivalent to using training examples in a smaller neighborhood to make predictions, and the prediction results will be very sensitive to the instance points of the neighbors. If the nearby instance points happen to be noise, the prediction will be wrong. In other words,$The\; reduction\; of\; k$ value means that the overall model becomes complex and prone to overfitting.
  If a larger$The\; k$ value is equivalent to using training examples in a larger neighborhood to make predictions. At this time, training examples that are far (dissimilar) from the input instance will also have an effect on the prediction, making the prediction wrong. $The\; increase\; of\; k$ value means that the overall model becomes simpler.
  In application,$The\; k$ value generally takes a relatively small value. Usually, the cross-validation method is used to select the optimal$k$ value.

3. Classification decision rules

The classification decision rule here is the majority vote, that is, $The\; majority\; class\; among\; the\; k$ adjacent training strengths determines the class of the input strength.

---

Three, $kd$ tree

  Through the above study, we have already understood $The\; specific\; steps\; of\; the\; k-$ nearest neighbor algorithm. At this time, you can use programming languages ​​such as python to realize$k$ nearest neighbor algorithm.
  However, if the dimensions of the instances in the training set are large and the training data capacity is large, if we calculate the Euclidean distance between the instance points in the training set and the prediction points one by one, and find out$Calculation\; is\; very\; time-consuming\; when\; there\; are\; k$ nearest neighbors, and this method is not feasible. In order to improve$For\; the\; efficiency\; of\; k-$ nearest neighbor search, you can consider using a special structure to store training data to reduce the number of distance calculations. There are many specific methods, here is one of$The\; method\; of\; k-d$ tree.

1. Construction balance $kd$ tree

  construct balance $k-d$ tree.

1. Start: Construct the root node, the root node contains $T$ 'skkA hyperrectangular region of$k-\; dimensional\; space.$select$x^1$ is the coordinate axis, withTTx 1 x^1of all instances in$T$$x^{The\; median\; of\; the\; 1}$ coordinates is the segmentation point, which divides the super-rectangular area corresponding to the root node into two sub-areas.
   Generate left and right child nodes with a depth of 1 from the root node: the coordinates of the left child node are$x^1$ is smaller than the sub-region of the segmentation point, and the right child node corresponds to the coordinate$x^1$ for subregions larger than the split point.
   Store the instance points falling on the segmentation hyperplane in the root node.
2. Repeat: for depth jjThe node of$j$$x^l$ is the split axis,$l=j\left(\text{mod}k\right)+1$ , take the xlx^lof all instances in the region of the node$x^{The\; median\; of\; l}$ coordinates is the segmentation point, and the super-rectangular area corresponding to the node is divided into two sub-areas.
   The depth generated by this node is$j+$The left and right child nodes of$1$$x^l$ is smaller than the sub-region of the segmentation point, the right child node corresponds to the coordinate$x^l$ is greater than the subregion of the cut point.
   Save the instance points falling on the segmentation hyperplane in this node.
3. Instructing that there are no instances in the two sub-regions is to stop, thus forming kd kdRegion partitioning of$k-$$d\; trees.$

We directly use an example to show how to construct $k-d$ tree.
Given a two-dimensional training data set
$$T=\{(2,3{)}^T,(5,4{)}^T,(9,6{)}^T,(4,7{)}^T,(8,1{)}^T,(7,2{)}^T\}$$
Construct a balanced$k-d$ tree.

1. In the first step, take $x^1$ is the dividing axis,$x^{The\; median\; in\; dimension\; 1}$ is 7, so we split the data with 7 as the split point. As shown in the figure below:
   
   At this point$(7,2)$ Points are stored at the root node (depth is 0), there are three points in the left area, and two points in the right area.
2. In the second step, the depth of the node is 1 at this time, $l=1\left(mod2\right)+1=2$ , choose$x^2$ is the coordinate axis of segmentation. x 2 x^2in the left area$x^{}$The median in${}^2$$x^{The\; median\; in\; the\; 2-}$ dimension is 6, and the left and right regions are segmented accordingly. As shown in the figure below:kd kd
   
   at this time$A\; k-d$ tree looks like this:
   
3. In the third step, the depth of the node at this time is 2, $l=2\left(mod2\right)+1=1$ , select$x^1$ is the coordinate axis of segmentation. After splitting, it is shown in the figure below:
   
   The kd tree is shown in the figure below:
   

So far we have found that there are no instances in all areas (that is to say, there are no instance points to divide), so stop kd kdRegion partitioning of$k-$$d\; trees.$

2. Use $k-d$ tree for nearest neighbor search

In the previous section, we have introduced how to construct $kd$ tree, in this section we introduce how to use$k-d$ tree for nearest neighbor search.
The algorithm is as follows:
Input: constructed$kd$ tree, target point$x$ .
Output:$The\; nearest\; neighbor\; of\; x$ .

1. Starting from the root node, recursively visit $k-d$ tree. If the target point$If\; the\; coordinates\; of\; the\; current\; dimension\; of\; x$ are smaller than the coordinates of the segmentation point, move to the left child node, otherwise move to the right child node until the child node is a leaf node.
2. Let this leaf node be the "current closest point" (compute point $x$ distance from this leaf node).
3. Back up recursively, at each node (denoted as $o$ ) Do the following:
   • (a) If the node $o$ Saved instance point with point$The\; distance\; of\; x$ is greater than the distance from the current closest point to point$x$ is closer, point$o$ as the "current closest point".
   • (b) The current closest point must exist in the node $oA$ region corresponding to a child node. Check the parent nodeooWhether there is a closer point in the area corresponding to another child node of$o\; .$Specifically, check whether the area corresponding to another child node is the same as the target point$x$ is the center of the sphere, and the hypersphere intersects with the distance between the target point and the "current closest point" as the radius. If it intersects, there may be a point closer to the target point in the area corresponding to another child node, and move to another child node. Next, recursively perform the nearest neighbor search, and if they do not intersect, back up.
4. When returning to the root node, the search ends. The final "current closest point" is xxThe nearest neighbor of$x\; .$

We use an example to demonstrate kd kdThe process of performing nearest neighbor search on$k$$d\; tree.$The target point is$x(2,4.5)$ , to find$The\; nearest\; neighbor\; of\; x$ .
According to the algorithm flow, after the first step, we move to$(4,7)$ point. The movement path is:$(7,2)-(5,4)-(4,7)$ .
The second step: let the leaf node$(4,7)$ is the current nearest point$o$ . Target point and closest point$The\; distance\; of\; o$ is 3.202.
The third step: recursively back up, back to$(5,4)$ point. (a) The distance between the target point and (5,4) is: 3.041, so$(5,4)$ The point is closest to the current point$oDistance$ from target point$x$ is closer, then update the current closest point$o$ is$(5,4)$ . (b) Take the target point$x$ is the center of the circle, and the target point and the "current closest point$o$ "The distance between the radius circle and the current closest point$o(5,4)$ another child node$(2,3)$ The corresponding regions intersect. So we move to$(2,3)$ points,$(2,3)$ The distance between the point and the target point is o ( 5 , 4 ) o(5,4)than the current closest point$o(5,4)$ is closer, so the nearest point o is updated to$(2,3)$ . Then continue to go back up to$(7,2)$ points,$(7,2)$ Point distance from target point ratio$(2,3)$ The point is far away from the target point.
Step 4: Go back to the root node, and the search ends. At this point the nearest point$o(2,3)$ is the target point$x(2,4.5)$ nearest neighbor point.

The overall idea is roughly;

1. First perform a binary search to find the leaf node
2. backtracking search path
3. Check if another child node region has a closer
4. search another subspace
5. Finally backtrack to the root node

4. Reference link

Code link: https://github.com/zgaxddnhh/lihang-code
Reference books: Machine Learning Methods. Li Hang