In many application fields, such as pattern recognition, semantic analysis, text classification, etc., it is usually high-dimensional data. In this case, dimensionality reduction is an efficient way to deal with these data. So far, many dimensionality reduction methods have emerged, which can be divided into three categories: unsupervised, supervised, and semi-supervised. Supervised dimensionality reduction methods mainly include Linear Discriminant Analysis (LDA), Marginal Fisher Analysis (MFA), and Maximum Margin Criterion (MMC). Unsupervised dimensionality reduction methods mainly include Principal Component Analysis (PCA), Locality Preserving Projection (LPP) and so on. The semi-supervised dimension reduction methods include semi-supervised dimension reduction (SSDR) and semi-supervised discriminant analysis (SDA). In this article I only talk about unsupervised dimensionality reduction methods. In my opinion, with the continuous development, the scale of data is getting bigger and bigger, the acquisition of supervised information is a problem, and unsupervised is the final winner. Supervised dimensionality reduction methods and semi-supervised methods will be written later.
Why dimensionality reduction
For a data matrix $X \in R^{n \times p}$, where $n$ represents the number of samples, $p$ represents the number of variables, that is, the dimension, for each variable is zero mean. In many data, the dimension $p$ is often very high. For example, in image data, we all know that an image is a matrix, assuming it is a square matrix $m \times m$. We expand the data matrix of an image into a row vector, so that multiple images are arranged in a row into a matrix, $n$ represents how many images there are, and $p=m \times m$ represents the dimension of the image. The dimension of such image data is very high, for example, an image with low resolution $256 \times 256$, when expanded into a vector, the dimension becomes 65536 dimensions, this dimension is very scary, and this is just a resolution lower image. So we need dimensionality reduction.
Principal Component Analysis (PCA)
Principal component analysis is one of the most popular methods of dimensionality reduction. Assuming that there is a data matrix $X \in R^{n \times p}$, we want to find a projection matrix that projects $X$ to a low-dimensional space, and makes the projected data samples have the largest variance, the so-called maximum Variance means that the samples can be separated as much as possible. Assuming that the projection direction vector is $W \in R^{p \times r}$, the matrix $X$ is projected to get $XW=Y$, $Y \in R^{n \times r}$ is the projected Matrix, where $r$ is the dimension of the data after dimensionality reduction, $r<p$, so the function of dimensionality reduction is achieved. Next, let's consider how to choose the projection matrix $W$, our objective function is to make the reduction The post-dimensional matrix $Y$ has the largest between-sample variance. So we get the following optimization problem:
$$\max \frac{w^{T}X^{T}Xw}{w^{T}w}$$
Solving this problem is equivalent to finding the largest eigenvalue of the matrix $X^{T}X$. If we solve the first $r$ eigenvalues, the eigenvectors corresponding to the first $r$ eigenvalues constitute a projection Matrix $W$. Principal component analysis can be applied to the eigenface problem.
Locality Preserving Projection (LPP)