Intuitive understanding - Mahalanobis distance

First of all, we have a good understanding of Euclidean distance, which is used to calculate the distance between two points in Euclidean space (that is, our common coordinate system).
For example point $x=(x_1,\dots ,x_n)$和$y=(y_1,\dots ,y_n)$ 的欧氏距离为：
$$\begin{gathered} d(x,y)=\sqrt{(x_1-y_1{)}^2+(x_2-y_2{)}^2+...+(x_n-y_n{)}^2} \\ =\sqrt{\sum _{i=1}^n(x_i-y_i{)}^2} \\ =\sqrt{(x-y{)}^T(x-y)} \end{gathered}$$

The Mahalanobis distance is mostly used in anomaly detection. As shown in the figure below, our data is distributed in an ellipse, and the center point of the ellipse is the green point in the middle. Now we need to judge whether the green point or the blue point outside is an outlier point, which one is a normal point?

It is obvious that the green points are inside the ellipse, so the green points should be normal points and the blue points should be outliers. But if calculated by Euclidean distance, blue should be a normal point. So there is a problem with the Euclidean distance at this time.

Let’s take another one-dimensional chestnut. Now there are two categories with uniform units. The first category has a mean of 1 and a variance of 0.1, and the second category has a mean of 5 and a variance of 4. Then a point with a value of 2.5 has a greater probability of belonging to the first category or the second category?

In terms of Euclidean distance, it should be the first type, but intuitively it is obviously the second type, because the value range of the first type is $1\pm 0.1$ , the value range of the second category is$5\pm 4$ , so it must be the second category.

See the clue yet? It's the mean and variance problem.

Before officially starting the Euclidean distance, I hope you can have the basis of PCA: PCA principal component analysis-clear, detailed and easy to understand . This article is written very clearly and in detail. It is recommended to draw for ten minutes to understand PCA.

From this article, we know that the data can be centered by first subtracting the mean value (the left of the figure), and then the base transformation is performed to map the original data to the data shown on the right of the figure below.

This is only intuitively more convenient, but still does not change the Euclidean distance between the blue and green points and the center point.

At this time, if you are careful, you must have observed the clue again. The variance of the data in the horizontal direction is relatively large, while the variance in the vertical direction is relatively small. A distribution with mean 0 and variance 1).
There are two dimensions here, so it is necessary to standardize both x and y directions.

$$x_{new}=\frac{x-}{p}$$

where μ is the sample mean and σ is the standard deviation of the sample data.

Here I randomly made some data

x = [0,-3,-3,-2,-2,-2,-1,-1,-1,-1,0,0,1,1,2,2,2,3]
y = [-1.5,-0.5,0.4,-0.5,-1,0.2,0.1,0,-1,0.5,1.2,-0.5,0.8,0.5,0,-0.5,0.5,0]

Visualize it as shown in the figure:

you can see $(0,-1.5)$ and$(3,0)$ , the intuitive feeling is$(3,0)$ is far away from the far point, which is an outlier point. But look at the distribution of the data again. The variance on the x-axis is large and the variance on the y-axis is small. Let’s look at the alignment and standardization:

The same coordinate system, after normalization and scaling, the original $(0,3)$ The point is closer to the origin.

import numpy as np
import matplotlib.pyplot as plt
x = [0,-3,-3,-2,-2,-2,-1,-1,-1,-1,0,0,1,1,2,2,2,3]
y = [-1.5,-0.5,0.4,-0.5,-1,0.2,0.1,0,-1,0.5,1.2,-0.5,0.8,0.5,0,-0.5,0.5,0]

x = np.array(x)
y = np.array(y)
plt.plot(x,y,'+')
plt.plot(0,0,'ro')
plt.plot(x[0],y[0],'ro')
plt.plot(x[-1],y[-1],'ro')
plt.xlim(-4,4)
plt.ylim(-3,3)
plt.show()

a = np.mean(x)
b = np.mean(y)

print(np.mean(x-a), np.mean(y-b))
print(np.std(x), np.std(y))

x1 = (x-a)/np.std(x)
y1 = (y-b)/np.std(y)

print(x1)
print(y1)
plt.plot(x1[0],y1[0],'ro')
plt.plot(x1[-1],y1[-1],'ro')
plt.xlim(-4,4)
plt.ylim(-3,3)
plt.plot(x1,y1,'+')
plt.plot(0,0,'ro')
plt.xlim(-4,4)
plt.ylim(-3,3)
plt.show()

From this PCA article we know: Let $C$ is forXXThe covariance matrix of$X$$Y$ is$X$ versus$P$ is the data matrix after base transformation, that is,$Y=PX$ , note$P$ is an orthogonal matrix. The base transformation performed above is$Y=The\; process\; of\; PX$ . After the transformation, there is Y, and then finding the Euclidean distance of Y is the so-called Mahalanobis distance.

So how to combine the above process in a unified way?

设$Y$ -shaped horizontal distance D, for:
$$\begin{array}{lll}D& =& \frac{1}{m}Y{Y}^{\mathsf{T}}\\ & =& \frac{1}{m}(PX)(PX{)}^T\\ & =& \frac{1}{m}PX{X}^{\mathsf{T}}{P}^T\\ & =& P(\frac{1}{m}X{X}^{\mathsf{T}})P^{\mathsf{T}}\\ & =& PC{P}^{\mathsf{T}}\end{array}$$

From PCA we know that the transformed $X$ is uncorrelated (orthogonal) in the new vector space, so D is diagonal. Due to$X$ is known, so its covariance matrix$C$ also knows, so how to find P to transform X?

Thanks to $C$ is a symmetric matrix. In linear algebra, a real symmetric matrix has a series of very good properties:

1. The eigenvectors corresponding to different eigenvalues ​​of a real symmetric matrix must be orthogonal.
2. Let the eigenvector $\lambda$ multiplicity is r, then there must be r linearly independent eigenvectors corresponding to$\lambda$ , so these r eigenvectors can be unit orthogonalized.

From the above two, it can be seen that a real symmetric matrix with n rows and n columns must be able to find n unit orthogonal eigenvectors, let these n eigenvectors be $e_1,e_2,\cdots ,e_n$, we organize it into a column-wise matrix:
$$E=\left(\begin{array}{cccc}{e}_1& e_2& \cdots & e_n\end{array}\right)$$
Equivalent matrix function C:
$$E^{\mathsf{T}}CE=L=\left(\begin{array}{cccc}{l}_1& & & \\ & l_2& & \\ & & \ddots & \\ & & & l_n\end{array}\right)$$

where $\Lambda$ is a diagonal matrix, and its diagonal elements are the eigenvalues ​​corresponding to each eigenvector (there may be repetitions).

The above conclusions no longer provide strict mathematical proofs. Friends who are interested in proofs can refer to the content of "diagonalization of real symmetric matrices" in linear algebra books.

At this point, we find that we have found the required matrix P:

$$P=E^{\mathsf{T}}$$

P is 协方差矩阵的特征向量单位化后按行排列出的矩阵where each row is an eigenvector of C. If P is set according to Λ \LambdaThe eigenvalues ​​in$\Lambda$用P的前K行组成的矩阵乘以原始数据矩阵X，就得到了我们需要的降维后的数据矩阵Y are arranged from large to small, and the eigenvectors are arranged from top to bottom, then .

I don’t know how to push the above, so I have the following

Original address: www.cnblogs.com

Mahalanobis Distance

Mahalanobis Distance (Mahalanobis Distance) was proposed by Indian statistician Mahalanobis (PC Mahalanobis), which represents the covariance distance of data . It is an efficient method to calculate the similarity between two unknown sample sets. It takes into account the connection between data features and is scale-invariant, i.e. independent of the measurement scale.

Definition of Mahalanobis distance

Assume $x$，$y$ is obtained from the mean vector as$\mu$ , the covariance matrix is​​Σ \SigmaOverallGG$of\; \Sigma$Two samples drawn at random from$G$$x$，$The\; Mahalanobis\; distance\; between\; two\; points\; in\; y$ is:

$$d_m^2(x,y)=(x-y{)}^T{\Sigma }^{-1}(x-y)$$

define $x$ vs. overall$The\; Mahalanobis\; distance\; of\; G$ is:

$$d_m^2(x,m_G)=(x-m_G{)}^T{\Sigma }^{-1}(x-m_G)$$

Among them, if the covariance matrix is ​​a unit vector, that is, each dimension is independently and identically distributed, the Mahalanobis distance becomes the Euclidean distance.

Note: The above two expressions are actually the square of the Mahalanobis distance

Why define Mahalanobis distance

1. The impact of the unit of the data index on the distance measurement

In many machine learning problems, the features between samples can be described by distance, such as the most common Euclidean distance. For Euclidean distance, any two points in space $P=(x_1,x_2,\dots ,x_p)$与$Q=(y_1,y_2,\dots ,y_p)$ is the distance between:

$$d(P,Q)=\sqrt{(x_1-y_1{)}^2+(x_2-y_2{)}^2+\cdots +(x_p-y_p{)}^2}$$

Obviously, when the fixed point $Q$ and the value is$When\; 0$ , it means point$P$ is the distance from the coordinate origin.

One disadvantage of Euclidean distance is that when each component is a quantity of different nature (for example, the height and weight of a person, the weight and volume of a watermelon), the size of the "distance" is actually related to the unit of the index. For example, horizontal axis $x_1$Represents weight (in $kgg$ ), the vertical axis$x_2$Represents the length (in $cm\; isthe$ unit). There are four points$A,B,C,D$ , as shown in Figure 2.1.

Figure 2.1

at this time

$d(A,B)=\sqrt{5^2+10^2}=\sqrt{125}$
$d(C,D)=\sqrt{10^2+1^2}=\sqrt{101}$

Obviously, $AB$ ratio$CD\; shouldbe$ longer.

Now, if $x_2$In millimeters $\left(mm\right)$ as unit,$x_1$remain unchanged, at this time $A$ coordinate$(0,50)$，$C$ coordinates are$(0,100)$ , then

$d(A,B)=\sqrt{50^2+10^2}=\sqrt{2600}$
$d(C,D)=\sqrt{100^2+1^2}=\sqrt{10001}$

Result $CD$ is better than$AB$ long! This is obviously unreasonable.

2. The influence of sample distribution on the distance measure

Although we can do normalization first to eliminate the problem of different measurement units between dimensions, the sample distribution will also affect the distance measurement. We can see that the weight of each coordinate in the Euclidean distance is equal. But in real problems, when the coordinate axes represent observation values, they often have random fluctuations of different sizes, that is, have different variances, and obviously the weights between them are different.

Here are two one-dimensional examples to illustrate this problem:

Assuming two normal distribution populations $G_1:N(m_1,p_1^2)$ and$G_2:N(m_2,p_2^2)$ . If there is a sample whose value is at point A, which population is A closer to? As shown in Figure 2.2:

Figure 2.2

In terms of absolute length, point A is $G_1$Closer, that is, point A to $m_1$Ratio to $m_2$To be closer, (the Euclidean distance is used here, and the coordinates of point A are compared with $m_1$to $m_2$the absolute value of the difference between values), but from a probability point of view, point A is at $m_1$About 4 $p_1$, while at $m_2$About 3 $p_2$, if measured by standard deviation, the distance from point A is $m_2$To be "closer". The latter is considered from the perspective of probability. It divides the square of the coordinate difference by the variance (or multiplies the reciprocal of the variance), thus turning it into a dimensionless number. When it is extended to multi-dimensional, it must be multiplied by the covariance matrix Σ \SigmaThe inverse matrix of$\Sigma$$S^{-1}$。

Look at Figure 2.3 below again, the distances between A and B relative to the origin are the same. But since the sample population is distributed along the horizontal axis, point B is more likely to be a point in this sample, and point A is more likely to be an outlier. A small difference in a dimension with small variance can become an outlier.

Figure 2.3

3. The influence of correlation between dimensions on distance measurement

We see that the situation described above is that the dimensions or features are not correlated, so what if there is correlation between the dimensions? As shown in Figure 2.4 below:

Figure 2.4

It can be seen that the sample point can be approximated as $f(x)=For\; the\; linear\; distribution\; of\; x$ , the distances between A and B relative to the origin are still the same. Obviously, A is more like an outlier, that is to say, it is farther away from the sample population.

Even if the data has been standardized, it will not change the relationship between AB and the distance between the origin. So to essentially solve this problem, it is necessary to standardize the principal components in the principal component analysis .

Why standardization does not change the relationship between distances , here is a brief explanation of the most common z-score standardization (also called standard deviation standardization) as an example. This method gives the mean (mean) and standard deviation (standard deviation) of the original data to standardize the data. The processed data conform to the standard normal distribution, that is, the mean is 0, the standard deviation is 1, and its transformation function is: $x^{\ast }=(x-\mu )/\sigma$，使用$\mu$ is the mean of all sample data,$\sigma$ is the standard deviation of all sample data.

Obviously here $\mu$和$\sigma$ is the same, sostandardization is only equivalent to a proportional scaling of the distance between points A and B and the data center, and does not affect the relationship between them.
It can be seen that there are still big problems only relying on the standardized Euclidean distance, and the influence of data correlation on the judgment result is still very large.

Why is the standard deviation of z-score normalized to 1?

x-μ only changes the mean, and the standard deviation remains unchanged, so the mean becomes 0
(x-μ)/σ only divides the standard deviation by σ times, so the standard deviation becomes 1, that is,
E
$And(x^{\ast })=E[\frac{x-E(x)}{\sqrt{D(x)}}]=\frac{1}{\sqrt{D\left(x\right)}}E[x-E(x)]=0$

$D(x^{\ast })=D\left[\frac{x-E\left(x\right)}{\sqrt{D\left(x\right)}}\right]=\frac{1}{D\left(x\right)}D[x-E(x)]=\frac{D(x)}{D\left(x\right)}=1$

OK! To sum up the question:

1) Consider the influence of correlation between dimensions
2) Consider the influence of variance
3) Consider the influence of measurement units or dimensions

Therefore, it is necessary to establish a statistical distance (just a term used to distinguish the commonly used Euclidean distance). The distance is required to be independent of the units used for each variable. It appears that the distance we choose depends on the variance and covariance of the samples. The Mahalanobis distance is the most commonly used statistical distance.

Geometric meaning of Mahalanobis distance

So what to do? , in fact, if the above content has been understood, we will know that we only need to rotate the variables according to the principal components, so that the dimensions are independent of each other , and then standardize, so that the dimensions have the same distribution , then it is OK, and then the calculation of the Mahalanobis distance becomes Became the calculation of Euclidean distance.

According to the principal component analysis, since the principal component is the direction of the eigenvector, the variance of each direction is the corresponding eigenvalue, so it is only necessary to rotate according to the direction of the eigenvector, and then scale the eigenvalue times. The following results can be obtained, Figure 2.5:

Figure 2.5

The outliers are successfully separated, and the Mahalanobis distance at this time is the Euclidean distance.

Derivation of the Mahalanobis distance

Suppose the original multidimensional sample data $X_{n\times m}$（$n$ lines,$m$ columns):

$$X_{n\times m}=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_1\\ x_{21}& x_{22}& \cdots & x_2\\ ⋮& ⋮& \ddots & ⋮\\ x_n& x_n& \cdots & x_{nm}\end{array}\right]$$

where each row represents a sample, $X_i$Indicates the $i$个维度，$X_i=(x_1,x_2,\dots ,x_{}{)}^T,i=1,2,\dots ,m$ , then the above multidimensional sample data can be recorded as:$X=(X_1,X_2,\dots ,X_m)$

The mean vector of the sample population is recorded as: $m_X=(m_X,m_{X2},\dots ,m_{Xm})$

The covariance matrix is ​​written as: $S_X=And[(X-m_X{)}^T(X-m_X)]=\frac{1}{n}(X-m_X{)}^T(X-m_X)$

According to the previous description, the data must first be transformed and rotated to the principal components to make the dimensions linearly independent. Suppose the original data $X$ through the coordinate rotation matrix$The\; U$ transformation has obtained new coordinates, and the corresponding new data set is recorded as$F$ (actually$X$，$F$ represents the same data set, but due to different coordinate values, different marks are used for distinction), data set$The\; mean\; vector\; of\; F$ is recorded as$m_F=(m_F,m_{F2},\dots ,m_{Fm})$ , then:

$$F^T=(F_1,F_2,\dots ,F_m)=U{X}^T$$

$$(F-m_F{)}^T=U(X-m_X{)}^T$$

$$(F-m_F)=(X-m_X)U^T$$

The transformed data dimensions are linearly independent, and the variance of each dimension is an eigenvalue, that is, the covariance matrix $S_F$is a diagonal matrix, so:

$$\begin{array}{rl}{S}_F& =E[(F-m_F{)}^T(F-m_F)]\\ & =\frac{1}{n}(F-m_F{)}^T(F-m_F)\\ & =\frac{1}{n}U(X-m_X{)}^T(X-m_X)U^T\\ & =The{S}_X{U}^T\\ & =\left(\begin{array}{cccc}{l}_1& & & \\ & l_2& & \\ & & \ddots & \\ & & & l_m\end{array}\right)\end{array}$$

where $l_i,i=1,2,\dots ,m$ represents the variance of each dimension.

After deriving here, we can derive the Mahalanobis distance formula, assuming that we want to calculate XXA sample point x of$X$$x=(x_1,x_2,\dots ,x_3)$到重心$m_X=(m_X,m_{X2},\dots ,m_{Xm})$ Mahalanobis distance. Equivalent to finding$F$中点 $f=(f_1,f_2,\dots ,f_m)$ to the coordinate value of the center of gravity of the standardized data$m_F=(m_F,m_{F2},\dots ,m_{Fm})$ Euclidean distance.

$$\begin{array}{rl}{d}_m^2(f,m_F)& =(\frac{f_1-m_F}{\sqrt{l_1}}{)}^2+(\frac{f_2-m_{F2}}{\sqrt{l_2}}{)}^2+\cdots +(\frac{f_m-m_{Fm}}{\sqrt{l_m}}{)}^2\\ & =(f_1-m_F,f_2-m_{F2},\dots ,f_m-m_{Fm})\left(\begin{array}{cccc}\frac{1}{l_1}& & & \\ & \frac{1}{l_2}& & \\ & & \ddots & \\ & & & \frac{1}{l_m}\end{array}\right)\left(\begin{array}{c}{f}_1-m_F\\ f_2-m_{F2}\\ ⋮\\ f_m-m_{Fm}\end{array}\right)\\ & =(f-m_F)(The{\Sigma }_X{U}^T{)}^{-1}(f-m_F{)}^T\\ & =(x-m_X)U^T(U{\Sigma }_X{U}^T{)}^{-1}U(x-m_X{)}^T\\ & =(x-m_X)U^T(U^T{)}^{-1}{S}_X^{-1}{U}^{-1}U(x-m_X{)}^T\\ & =(x-m_X)S_X^{-1}(x-m_X{)}^T\end{array}$$

This is the calculation formula of the Mahalanobis distance defined earlier

if $x$ is a column vector, then

$$d_m^2(f,m_F)=(x-m_X{)}^T{\Sigma }_X^{-1}(x-m_X)$$

If the center of gravity in the derivation process $m_X$Change to any other sample point $y$ , then the Mahalanobis distance formula between two sample points can be obtained as:

$$d_m^2(x,y)=(x-y{)}^T{\Sigma }_X^{-1}(x-y)$$

Resort $E$ represents a point set,$d$ represents the distance, it is$E\times E$ to$[0,\infty )$ , it can be proved that the Mahalanobis distance conforms tothe basic properties of the distance measure

that's all!

Machine Learning Notes - Distance Measure and Similarity
1) Machine Learning Notes - Distance Measure and Similarity (1) Minkowski Distance

Reference sources:
1) https://www.jianshu.com/p/5706a108a0c6
2) https://blog.csdn.net/u010167269/article/details/51627338
3) https://zhuanlan.zhihu.com/p /46626607
4) Multivariate Statistical Analysis- He Xiaoqun

I really want to scold csdn, but I can’t understand it after searching for a long time, so I understand it after reading this article in the blog garden. . .
After all, it's your own food!