0. Background
Knowledge discovery in many aspects of search technology relies on eigenvalue or singular value problems, involving eigenvalue calculation problems.
There is no direct way to compute eigenvalues.
The calculation method of positioning eigenvalues is based on the idea of power iteration, which is a kind of iterative method for solving eigenvalues. A sophisticated version of this idea is known as the QR algorithm and is a general method for determining all eigenvalues of a typical matrix.
If the eigenvalue problem is reduced to the problem of finding the roots, using a root solver to find the roots of a polynomial with a slight error will bring disastrous consequences.
1. Power iteration method
The main idea of the power iteration method: the eigenvector corresponding to the dominant eigenvalue will be dominant in the calculation process after multiple calculations.
Main methods: normalization and multiplication with matrix A.
As the iteration continues to improve the approximate eigenvector, how to get the eigenvalue? Rayleigh quotient.
Limitations of the power iteration method: limited to solving the eigenvalue with the largest absolute value.
Inverse of Power Iteration: If Power Iteration is used for the inverse of a matrix, the smallest eigenvalue can be found.
Now that we know how to find the maximum or minimum of a matrix, what about other eigenvalues?
The power iteration method gives a solution here:
In order to find out the eigenvalues of the matrix A near the real number s, use the power iteration method for (A-sI)^-1 to get the maximum eigenvalue b of (A-sI)^-1, then x=b^-1+s is the eigenvalue of the matrix A around the real number s.
2. QR algorithm
The QR algorithm is a method that can find all eigenvalues at once.
QR decomposition is to decompose the matrix into the product of an orthogonal matrix and an upper triangular matrix. There are many methods for the actual calculation of QR decomposition, such as Givens rotation, Householder transformation, and Gram-Schmidt orthogonalization, etc.
set up
for j=1,2,3...
end
The above process can be called normalized simultaneous iteration. In step j, the obtained
column is the approximation of the eigenvector of A, and the diagonal elements
are the approximate eigenvalues.
This algorithm becomes a mobile-free QR algorithm, but it has strict requirements on the matrix A (that is, A is a symmetric
matrix and satisfies the eigenvalues
), even if A is a symmetric matrix and does not satisfy the conditions of the theorem, it may fail. And the algorithm is difficult to modify to calculate complex eigenvalues, and the iteration speed is relatively slow.
Therefore, we need to make a set of improvements to make the eigenvalue calculation more general and accelerate the convergence.
3. Heisenberg
The main idea of Heisenberg is to replace A with a similarity matrix in the form of Heisenberg, i.e. apply a similarity transformation before QR iterations to place more 0s in A while maintaining the eigenvalues. Additionally, Heisenberg will eliminate the problem of QR iterations failing to converge to complex features.
The Heisenberg matrix form is shown in the figure below:
So how to transform matrix A into Heisenberg form? Householder transformation.
Householder transformation
The Householder transformation can change some elements of a vector to zero while maintaining the norm of the vector.
By multiplying the reflector H on the left and right sides of the matrix A step by step, an eigenmatrix with the same eigenvalues is obtained.
The result of doing 3 steps of operation:
On the left is the 5x5 Heisenberg matrix we got. Since this matrix is similar to A, it has the same eigenvalues and multiplicity of eigenvalues as A.
Generally, for an nxn matrix A, n-2 Householder steps are required to convert A into Heisenberg form.
4. Mobile QR algorithm
The mobile QR algorithm uses the technique of power iteration to greatly accelerate the convergence speed.
In order to reduce the amount of calculation, the Householder matrix is generally used to transform the matrix A into a quasi-upper triangular matrix (Haisenberg form) , and then the eigenvalues
are calculated by the QR method of two-step displacement .
The specific algorithm process of finding all the eigenvalues of matrix A using the QR method of double-step displacement is as follows:
5. Gaussian elimination and LU decomposition
Gaussian elimination
Gaussian elimination is a useful tool for solving linear equations of appropriate size, efficiently solving n equations with n unknowns. Gaussian elimination is mainly composed of two unequal parts, the elimination process with relatively large computational cost, and the back-substitution process with relatively small computational cost. The elimination calculation of n equations and n unknowns can
be completed after one operation.
LU decomposition
LU decomposition is the matrix form of Gaussian elimination, which involves writing the coefficient matrix A as the product of the lower triangular matrix L and the upper triangular matrix U. Among them, the U matrix is the upper triangular matrix obtained by the traditional Gaussian elimination process, and the corresponding L matrix is obtained by placing 1 on the main diagonal, and then placing the multipliers in the lower triangular matrix according to their specific positions during elimination.
How to use LU decomposition to convert back to the generation step?
Ax=b ---> LUx=b --->c=Ux Lc=b find c---> Ux=c find x
However, not all matrices can be decomposed by LU, we need to do some work before LU decomposition: some pivots.
partial pivot
The idea of partial pivot is to find the largest element in the first column before each elimination step, and exchange its corresponding row with the pivot row, that is, PA=LU.
The advantage of the Gaussian pivot method is that it has a process of selecting the pivot, which can avoid the situation that the program selects the principal element as 0 when performing the elimination operation, and also reduces the rounding error of the calculation, thereby improving the program. generalizability and accuracy of results.