Machine Learning Fundamentals: K Nearest Neighbor Algorithm (Machine Learning Fundamentals: KNN)

basic concept

The k-nearest neighbor (k-NN) method is a basic classification and regression method proposed by Cover T and Hart P in 1967.

The basic concept is as follows: there is a sample data set, all feature attributes are known, and each object in the sample set has a known category. For the object to be tested that does not know the classification, each feature attribute of the object to be tested is compared with the corresponding feature attribute of the data in the sample set, and then the algorithm extracts the classification label of the most similar object (nearest neighbor) of the sample. Generally speaking, we only select the top k most similar object data in the sample data set, which is the source of k in the k-nearest neighbor algorithm, usually k is an integer not greater than 20. Finally, according to the characteristics and attributes of the k data, the classification of the data to be tested is judged

Right now:一个样本与数据集中的k个样本最相似， 如果这k个样本中的大多数属于某一个类别， 则该样本也属于这个类别。

distance function

Manhattan distance (L1 norm)

Manhattan distance is also called "Manhattan block distance". Imagine you are on the streets of Manhattan, driving from one intersection to another intersection, the driving distance is this "Manhattan distance".

• Two points $a(x_1,y_1),b(x_2,y_2)$ between the Manhattan distance formula:
  $$d_{ab}=\left|x_1-x_2\right|+|y_1-y_2|$$
• Two points $a(x_1,x_2...x_n),b(y_1,y_2...y_n)$ Manhattan distance formula:
  $$d_{ab}=\left|x_1-y_1\right|+|x_2-y_2|+\dots ..+|x_n-y_n|$$

Euclidean distance (L2 norm)

• Two points $a(x_1,y_1),b(x_2,y_2)$ between Euclidean distance formula:
  $$d_{ab}=\sqrt{{\left(x_1-x_2\right)}^2+{\left(y_1-y_2\right)}^2}$$

• Two points $a(x_1,x_2...x_n),b(y_1,y_2...y_n)$的欧氏距离公式：
  $$d_{ab}=\sqrt{{\left(x_1-y_1\right)}^2+{\left(x_2-y_2\right)}^2+\dots +{\left(x_n-y_n\right)}^2}$$

Chebyshev distance ( $\infty$ norm)

• Two points $a(x_1,y_1),b(x_2,y_2)$ between Chebyshev distance formula:
  $$d_{ab}=\max \left(\left|x_1-x_2\right|,\left|y_1-y_2\right|\right)$$

• Two points $a(x_1,x_2...x_n),b(y_1,y_2...y_n)$的切比雪夫距离公式：
  $$d_{ab}=\max \left(\left|x_1-y_1\right|,|x_2-y_2|,\dots |x_n-y_n|\right)$$

Angle cosine

• Two points $a(x_1,y_1),b(x_2,y_2)$ between the angle cosine formula:
  $$\cos Th=\frac{x_1\ast x_2+y_1\ast y_2}{\sqrt{x_1^2+y_1^2}\ast \sqrt{x_2^2+y_2^2}}$$

• Two points $a(x_1,x_2...x_n),b(y_1,y_2...y_n)$的夹角余弦公式：
  $$\cos Th=\frac{x_1\ast y_1+x_2\ast y_2+\dots +x_n\ast y_n}{\sqrt{x_1^2+x_2^2+\dots +x_n^2}\ast \sqrt{y_1^2+y_2^2+\dots +y_n^2}}$$

KD-Tree

Since the KNN algorithm needs to calculate the nearest K samples in the test sample and the training sample, the time cost of the linear search is too large. Therefore, the commonly used search method is KD-Tree, whose basic principle is similar to binary search tree (BST), except that the feature dimension is higher. The schematic diagram is as follows:

The details are not mentioned here, you can refer to [1][2].

The KD-Tree algorithm is provided in sklearn.

Advantages and disadvantages

advantage

• Simple and easy to use, easy to understand, high precision, mature theory, can be used for both classification and regression;
• Can be used for non-linear classification
• Can be used for numerical data and discrete data (both for valuation and classification)
• Training time complexity is O(n); no data input assumptions;
• Insensitive to outliers.
• High accuracy, no assumptions about data, not sensitive to outlier;

shortcoming

• High computational complexity; high space complexity; requires a lot of memory
• Sample imbalance problem (i.e. some classes have a large number of samples while others have few samples);
• Generally, this is not used when the value is large, and the amount of calculation is too large. But a single sample should not be too small, otherwise it is easy to misclassify.
• The biggest disadvantage is that it cannot give the intrinsic meaning of the data.

think

references

1. The principle of KD Tree and its implementation in Python
2. Explain KDTree in detail
3. Detailed Explanation of KNN (k Nearest Neighbor) Algorithm for Machine Learning