Time university study linear algebra, eigenvalues (eigenvalue) and a feature vector (eigenvector) has been less understanding, though that textbooks eigenvalues and vector technology has been widely applied in the field of engineering, but in addition to know how to solve the eigenvalues and vector addition, little is known about its practical significance contained. After graduate study statistics, performing principal component analysis, eigenvalues and eigenvectors need to solve the covariance matrix of variables, and to determine the principal component of characteristic values according to the size, it seems to know a little bit of reality eigenvalues and eigenvectors meaning, but in line with the examination-oriented attitude, do not go in-depth understanding of the eigenvalues and eigenvectors. Some methods recently read machine learning, when such feature reduction methods such as SVD and the PCA, linear discriminant method (Linear Discriminant Analysis, LDA) or the like are related to the eigenvalues and eigenvectors, found if not in-depth understanding eigenvalues and vector, learning these methods can only float on the surface, it is difficult to understand thoroughly. Bitter experience, decided to learn about the outside to the inside good understanding and representation of eigenvalues and eigenvectors, this article about the eigenvalues and eigenvectors of a large number of online reference information, only as I study notes, declined to reprint.
First, the concept and the calculated eigenvalues and eigenvectors
Look at the definition of the textbook: set A is n -order square matrix, if there exists a constant 
and a nonzero n vector X , so that 
, called 
a matrix A eigenvalues, X is A belonging eigenvalues 
eigenvectors. Given n -order matrix A , the determinant
The result is about 
a polynomial matrix A becomes characteristic polynomial , polynomial equation of the characteristic configuration of 
the characteristic equation of the matrix A is called.
Theorem: n n eigenvalues order matrix A which is the characteristic equation 
of n heel 
; belonging to the eigenvalues of A 
feature vector is followed by a linear equation 
nonzero solution.
Example: find 
Eigenvalues and eigenvectors
Solution:
, solving a quadratic equation can get 
, 
;
X is the feature vector corresponding to satisfy 
, to obtain
X is the feature vector corresponding to satisfy 
, to obtain
Second, the geometric meaning of eigenvalues and eigenvectors
1, matrix transformation matrices, vectors, vector
Before making feature geometric meaning and interpretation of feature vectors, we first review the vectors , matrices , vector matrix transformation and other related knowledge.
Vector has row vectors and column vectors , the vector is to be interpreted in a geometric series of axis-parallel displacement, in general, any vector v can be written as " expansion " in the form:
In 3-dimensional vector as an example, the definition of P , Q , R & lt point to the + x, + y and + z direction unit vector , there are X = V P + Y Q + z R & lt . Now the vector v can be expressed as P , Q , R & lt 's linear transformation a. Where the basis vectors are Cartesian product axis , a fact that any coordinate system may be defined by three basis vectors, basis vectors as long as three linearly independent line (not on the same plane). Thus, a matrix is multiplied with a vector, such as Ax, expressed as follows:
If the rows of the matrix construed as basis vectors of a coordinate system, a vector matrix multiplication (or vector matrix multiplication) corresponding to perform a coordinate conversion, Ax of may be expressed as Y = x by the matrix A becomes transformed Y . Therefore, the origin of the traceability matrix, vector relationship with, we feel that the matrix is not a mystery, it's just in a compact way to express mathematical operations required for coordinate conversion.
2, eigenvalues and eigenvectors
Eigenvalues and eigenvectors of matrix A respectively 
and x, referred to as 
the expression is understood in the geometric space vector x through the matrix A obtained after transformation vector 
. This indicates that the vector x after transformation matrix A, no change in the direction (reverse direction is not the direction of change), only the telescopic 
fold.
Matrix 
, for example, the values of which 
, 
corresponding to the feature vector 
, 
then 
( ) 
represents a vector 
after the transformation matrix A to give 
, vector transform becomes changes 
direction, the knowledge 
expanded 2 times in the original direction. Characteristic value 
is the same reason, after the transformation matrix A eigenvectors 
expanded 3-fold in the original direction.
Thus, the basis vectors as feature vectors, matrix math is these groups to the corresponding feature value vectors required stretch. Given a matrix, we can find out the corresponding group (feature vectors), and through transformation vector (matrix), extending these groups (characteristic value).
Third, the application examples of eigenvalues and eigenvectors of
1, principal component analysis (Principle Component Analysis, PCA)
(1) variance, covariance, correlation coefficient, covariance matrix
variance:
Covariance: 
, 
, 
** univariate variance is a measure of the degree of dispersion, the covariance is a measure of the degree of correlation of two variables (closeness), the larger a covariance showed more similar the two variables (close), the smaller the mutual covariance between the two variables the greater degree of independence.
The correlation 
coefficient: ,
** covariance and correlation coefficient indicates that the two can measure the degree of correlation, covariance does not eliminate the dimension, the size of the covariance between the different variables can not be directly compared, and the correlation coefficient eliminates dimension, you can compare between different variables the degree of correlation.
Covariance matrix: If there are two variables X, Y, then the covariance matrix 
, the covariance matrix described Relationships between the variables in the sample.
Ideas and algorithms (2) Principal Component Analysis
Principal component analysis was performed with the idea of dimension reduction, a plurality of variables into a few integrated variables (i.e., main component), wherein each of the main component is a linear combination of the original variables, among the principal components unrelated to the main component of most of the information to reflect the beginning of the variable, and the information contained not overlap. It is a linear transformation, this transformation transforming the data to a new coordinate system such that any projection of the data of the largest variance in the first coordinate (referred to as the first principal component), the second largest variance in the second coordinate (second main component), and so on. The most important feature variance principal component analysis is often used to reduce the dimensionality of the data set, while maintaining the data set contribution.
Suppose that p variables described object, respectively X- . 1 , X- 2 ... X- p represented, this p variables consisting of p -dimensional random vector for the X-= (X- . 1 , X- 2 ... X- p ), n-samples configuration 
composed of n rows p matrix column A. Solving the main component as follows:
The first step in solving the obtained covariance matrix A B;
The second step, solving the covariance matrix B, obtained feature value vector is arranged in order of size 
, 
as a feature value vector of 
each of the feature values of a diagonal matrix consisting, U is a matrix of all the eigenvalues corresponding to eigenvectorsU, so there 
. Focus here , U are eigenvectors definite matrix, each row may be regarded as a vector of basis vectors, basis vectors of these matrices B after conversion obtained in each of the telescopic basis vectors, that is the size of the telescopic Feature vector.
A third step of selecting the number of main components, according to the size of the feature value, the feature value is larger as the main component, feature vectors corresponding to a vector-based screening feature value according to the actual circumstances, i.e. typically greater than 1 It can be considered as a main component.
(3) Case Study - Classification Machine Learning
Machine learning classification, giving 178 wine samples, each sample containing 13 parameters, such as alcohol, acidity, magnesium content, etc., these samples belong to three different types of wine. Task is to extract features three kinds of wine, so that the next sample is given a new wine when new samples can determine what kind of wine based on the existing data.
A detailed description of the problem: http: //archive.ics.uci.edu/ml/datasets/Wine
training data: http: //archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data
The data set is assigned a matrix of 178 rows R 13, and its covariance matrix C is a matrix of 13 rows 13 columns, the decomposition of the feature C, diagonalization 
, where U is a matrix consisting of eigenvectors, D is characteristic the composition of the diagonal matrix, press descending order. Then, so that 
, to achieve a set of data in Patent eigenvector orthogonal basis that the projection on. Ah, the focus here, 
the data are arranged in the column corresponding to the magnitude of the characteristic value, the latter corresponding to the smallest eigenvalue column, after removing the influence on the entire data set is relatively small. For example, we now directly remove seven behind, retaining only the first six, he completed dimensionality reduction.
Here we see a drop down front and classification results using svm dimension dimension of this part is achieved SVM R language pack e1071, the code shown in the following table. Classification results show, the use of principal component analysis of samples and classification results of principal component analysis as a sample is not performed. Therefore, the main component of the extracted principal components 6 can preferably 13 expression analysis of a sample of the original variables.
Library ( " e1071 " ) # read data wineData `read.table '= ( " E: \\ \\ Research in Progress Baidu cloud synchronization disc \\ blog \\ eigenvalues and eigenvectors of the data.csv \\ " , header = T , On Sep = " , " ); # Covariance matrix covariance = CoV (wineData [2:14 ]) # Calculating eigenvalues and eigenvectors eigenResult = Eigen (covariance) # Select six principal components, and calculates the six principal components of the sum of the variance explained PC_NUM. 6 = varSum = SUM (eigenResult values $ [. 1: PC_NUM]) / SUM ($ eigenResult values) # Drop sample dimensionality ruduceData = data.matrix (wineData [2:14])% *% $ eigenResult Vectors [,. 1 : PC_NUM] # Add labels # FinalData = cbind (wineData $ class, ruduceData) #给finalData添加列名 #colnames(finalDat) =c("calss","pc1","pc2","pc3","pc4","pc5","pc6") # Training samples - samples after principal component analysis as training samples Y = wineData $ class ; x1=ruduceData; model1 <- svm(x1, y,cross=10) predl <- Predict (MODEL1, X1) # predl <- Fitted (MODEL1) table (predl, Y) # using the table to see forecasts # Training samples - of the original data as the training sample X2 = wineData [2:14 ] model2 <- svm(x2, y,cross=10) #pred2 <- predict(model2, x2) pred2 <- fitted(model2) table (pred2, the y-) # use the table to see forecasts
【reference】