Traditional linear dimensionality reduction methods, such as principal component analysis (the PCA), factor analysis (FA) and the like, the variance of the sample concerned, can learn linear manifold structure, but can not learn nonlinear manifold. Although the method of classical manifold learning to learn nonlinear manifold structure, but because of their belonging transductive, a new sample can not be generalized. Some additional dimensionality reduction based on the kernel function, such as on KPCA, although nonlinear problem, but ignoring the manifold nonlinear structure.
NPE as locally linear embedding (the LLE) algorithm for linear approximation, it is possible to capture not only the structure of the nonlinear manifold, also retains the properties of a linear, a new sample can be generalized. Therefore, NPE in satisfactory, it is also possible to easily cope with the new sample, it has been widely used in industry.
As manifold problems, such as manifolds what, why learn manifolds, you can read my previous blog "manifold learning," the beginning of knowledge .
NPE algorithm ideas and steps
NPE and the same idea of the LLE, mainly local linear manifold structure remains unchanged dimensionality reduction process, thereby extracting useful information data. Here local linear structure is characterized by a weight matrix reconstruction, the weight matrix is reconstructed neighbor points of the neighborhood nodes reconstructed coefficients of the linear matrix.
And other classic manifold learning algorithm similar, NPE steps of the algorithm is divided into three steps:
1. Construction of Neighborhood Graph
A K nearest neighbor (the KNN) algorithm to construct neighborhood FIG. Suppose there are m samples, the m neighbor nodes common to the figures, wherein \ (\ mathbf {X} {i} \) represents the i-th node. If \ (\ mathbf {X} {J} \) is one of the K-nearest neighbor \ (\ mathbf {x} _ {i} \) , then connecting these two points, the converse is not connected.
2. Calculate the weight matrix (feature extraction manifold)
Weight matrix is provided for the \ (\ mathbf {W} \), where the element \ (W _ {ij} \) representative of the weight of the edge between node i and node j heavy, if there is no change between the two points, the corresponding the matrix elements to zero. Matrix \ (\ mathbf {W} \) a major element value by minimizing the following objective function is obtained:
\[ \min \sum _{i} \left| \left| \mathbf{x} _{i} - \sum _{j} W _{ij} \mathbf{x} _{j} \right| \right| ^{2}, \]
Wherein \ (\ mathbf {W} \) should satisfy the normalization constraint:
\[ \sum _{j} W _{ij} = 1,j=1,2,...,m. \]
3. Calculate the map
Dimension reduction is calculated by solving a linear mapping of the generalized eigenvector problem:
\[ \mathbf{XMX} ^{T} \mathbf{a}=\lambda \mathbf{XX} ^{T} \mathbf{a}, \]
Wherein the data set \ (\ X-mathbf {} = (\ mathbf {X} {}. 1, ..., \ mathbf {X} {m}) \), matrix \ (\ mathbf {M} = (\ mathbf { I} - \ mathbf {W} ) ^ {T} (\ mathbf {I} - \ mathbf {W}) \), matrix \ (\ mathbf {I} = diag (1, ..., 1) \) .
According to a feature value of the order from small to large (\ (\ the lambda {0} \ Leq ... \ Leq \ the lambda {}. 1-D \)) solving the eigenvectors are arranged into \ (\ mathbf {A} {0 }, ..., \ mathbf {a} {}. 1-D \), so that the embedding dimension reduction compared coordinates:
\[ \mathbf{y} _{i} = \mathbf{A} ^{T} \mathbf{x} _{i}, \]
among them
\[ \mathbf{A} = ( \mathbf{a} _{0}, \mathbf{a} _{1},...,\mathbf{a} _{d-1} ). \]
NPE derivation algorithm
NPE algorithm is the first major stream extracting local linear shape, and dimensionality reduction is achieved by keeping such information. Specifically, first with the NPE local linear reconstruction represented manifold partial linear structure, in the form of mean square error. Number of samples is set to m, the dimension is n, the number of fall of Wei Houwei d, \ (\ mathbf {Q} (i) \) k nearest neighbors to a set of samples in the sample i, the high-dimensional space characterized in weight configuration error objective function is:
\[ \phi ( \mathbf{W} ) = \sum ^{m} _{i=1} \left| \left| \mathbf{x} _{i} - \sum _{j\in \mathbf{Q}(i)} W _{ij} \mathbf{x} _{j} \right| \right| ^{2}, \]
Optimizing an objective function to obtain a matrix \ (\ mathbf {W} \) will contain information about the local manifold. In the process of dimension reduction of, NPE in the parameter space (low-dimensional space after the dimensionality reduction) to retain the same high-dimensional original space locally linear reconstruction of the target (i.e., using the same weight in the target function weighting matrix \ (\ mathbf {W} \)), to achieve a linear dimensionality reduction, low-dimensional objective function space is:
\[ \Phi ( \mathbf{y} ) = \sum ^{m} _{i=1} \left| \left| y _{i} - \sum ^{m} _{j=1} W _{ij} y _{j} \right| \right| ^{2}. \]
NPE derivation algorithm is divided into two steps, first computing the weight matrix, and the second is a low dimensional embedding calculation map. DETAILED derived as follows:
First, the first objective function is represented as a matrix form:
\[ \phi ( \mathbf{W} ) = \sum ^{m} _{i=1} \left| \left| \mathbf{x} _{i} - \sum _{j\in \mathbf{Q}(i)} W _{ij} \mathbf{x} _{j} \right| \right| ^{2} \]
\[ = \sum ^{m} _{i=1} \left| \left| \sum _{j\in \mathbf{Q}(i)} W _{ij} \mathbf{x} _{i} - \sum _{j\in \mathbf{Q}(i)} W _{ij} \mathbf{x} _{j} \right| \right| ^{2} \]
\[ = \sum ^{m} _{i=1} \left| \left| \sum _{j\in \mathbf{Q}(i)} W _{ij} ( \mathbf{x} _{i} - \mathbf{x} _{j} ) \right| \right| ^{2} \]
\[ = \mathbf{W} ^{T} _{i} ( \mathbf{x} _{i} - \mathbf{x} _{j} )( \mathbf{x} _{i} - \mathbf{x} _{j} ) ^{T} \mathbf{W} _{i} \]
\[ = \sum ^{m} _{i=1} \mathbf{W} ^{T} _{i} \mathbf{Z} _{i} \mathbf{W} _{i} \]
Wherein \ (\ mathbf {W is} {i} \) is the column vector the k-th nearest neighbor point samples i corresponding weighting factor consisting of a matrix \ (\ mathbf {the Z} {i} = (\ mathbf {X} { I} - \ mathbf {X} {J}) (\ mathbf {X} {I} - \ mathbf {X} {J}) ^ {T}, J \ in \ mathbf {Q} (I) \).
Considering the normalization constraints, we then be bound into a matrix equation form:
\[ \sum _{j\in \mathbf{Q}(i)} W _{ij} = \mathbf{W} ^{T} _{i} \mathbf{1} _{k} =1, \]
Wherein \ (\ mathbf {1} _ {k} \) is the k-dimensional vector of all ones.
Using the method of Lagrange multipliers to obtain the matrix equation objective function and constraints into one goal:
\[ \mathbf{L} (\mathbf{W}) = \sum ^{m} _{i=1} \mathbf{W} ^{T} _{i} \mathbf{Z} _{i} \mathbf{W} _{i} + \lambda'(\mathbf{W} ^{T} _{i} \mathbf{1} _{k} - 1). \]
For \ (\ mathbf {W} \) and allowed the derivation is 0, then:
\[ \mathbf{W} _{i} = -\dfrac{1}{2} \lambda \mathbf{Z} ^{-1} _{i} \mathbf{1} _{k}. \]
Using a normalization constraint, the result is normalized, the final weight matrix \ (\ mathbf {W} _ {i} \) is:
\[ \mathbf{W} _{i} = \dfrac{-\dfrac{1}{2} \lambda \mathbf{Z} ^{-1} _{i} \mathbf{1} _{k}}{\mathbf{W} ^{T} _{i} \mathbf{1} _{k}} \]
\[ = \dfrac{\mathbf{Z} ^{-1} _{i} \mathbf{1} _{k}}{\mathbf{1} ^{T} _{k} (\mathbf{Z} ^{-1} _{i}) ^{T} \mathbf{1} _{k}}. \]
After obtaining the weight matrix, since we can calculate the dimension reduction mapped. First define:
\[ z _{i} = y _{i} - \sum ^{m} _{j=1} W _{ij} y _{j}, \]
And then converted to a matrix form:
\[ \mathbf{z} = \mathbf{y} - \mathbf{Wy} = (\mathbf{I} - \mathbf{W}) \mathbf{y}. \]
The second objective function reduces to:
\[ \Phi(\mathbf{y}) = \sum ^{m} _{i=1} \left( y _{i} - \sum ^{m} _{j=1} W _{ij} y _{j} \right) ^{2} \]
\[ = \sum ^{m} _{i=1} (z _{i}) ^{2} \]
\[ = \mathbf{z} ^{T} \mathbf{z} \]
\[ = \mathbf{y} ^{T} (\mathbf{I} - \mathbf{W}) ^{T} (\mathbf{I} - \mathbf{W}) \mathbf{y}. \]
Since NPE linear dimensionality reduction algorithm, so the embedding dimension reduction can be expressed as:
\[ \mathbf{y} ^{T} = \mathbf{a} ^{T} \mathbf{X}, \]
Here \ (\ mathbf {y} \) m samples down to just one-dimensional embedding coordinates, \ (\ mathbf {a} \) is the corresponding map.
Then the second objective function can be further expressed as:
\[ \Phi(\mathbf{y}) = \mathbf{a} ^{T} \mathbf{X} (\mathbf{I} - \mathbf{W}) ^{T} (\mathbf{I} - \mathbf{W}) \mathbf{X} ^{T} \mathbf{a} \]
\[ = \mathbf{a} ^{T} \mathbf{X} \mathbf{M} \mathbf{X} ^{T} \mathbf{a}. \]
And embedding space coordinates also have constraints:
\[ \mathbf{y} ^{T} \mathbf{y} = 1 \implies \mathbf{a} ^{T} \mathbf{X} \mathbf{X} ^{T} \mathbf{a}. \]
Solving a same objective function, using the method of Lagrange multipliers where the second objective function and constraints were combined, and its derivative, can be obtained following the generalized eigenvector problem:
\[ \mathbf{XMX} ^{T} \mathbf{a}=\lambda \mathbf{XX} ^{T} \mathbf{a}. \]
Wherein \ (\ mathbf {M} \) may be calculated weight matrix \ (\ mathbf {W} \) stars, the above generalized eigenvector can solve. Finally, solve for the eigenvectors, the mapping matrix is arranged in order.
Code
Here we are given the python and matlab code.
python
First create a typical "Swiss roll" data sets:
import numpy
def swissroll(N=1000):
tt = numpy.array((5*numpy.pi/4)*(1+2*numpy.random.rand(N)))
height = numpy.array((numpy.random.rand(N)-0.5))
noise = 0.0
X = numpy.array([(tt+noise*numpy.random.randn(N))*numpy.cos(tt), 10*height, (tt+noise*numpy.random.randn(N))*numpy.sin(tt)])
return XThen, re-use shogun library to achieve NPE algorithm:
import modshogun as sg
# load data
feature_matrix = swissroll()
# create features instance
features = sg.RealFeatures(feature_matrix)
# create Neighborhood Preserving Embedding converter instance
converter = sg.NeighborhoodPreservingEmbedding()
# set target dimensionality
converter.set_target_dim(2)
# set number of neighbors
converter.set_k(10)
# set number of threads
converter.parallel.set_num_threads(2)
# set nullspace shift (optional)
converter.set_nullspace_shift(-1e-6)
# compute embedding with Neighborhood Preserving Projections method
embedding = converter.embed(features)matlab
As matlab code of Professor He Xiaofei home page on there, because of their length, there is not posted out.