[Linear Algebra 07] Markov Matrix and Fourier Matrix

  This article can be seen as an example of the application of determinants and eigenvalues. But I'll talk about the parts I'm interested in, namely discrete source channel models and diagonalization of circulant matrices.

Markov matrix

This matrix is ​​derived from the definition of probability in probability theory, so each element is actually a non-negative probability value . Markov matrix (Markov matrix), also known as probability matrix (probability matrix), transition probability matrix (transition probability matrix), the convention in English mathematics literature is to use the row vector of probability and the right random matrix of probability , so refer to Wikipedia The definition given above:

A random matrix describes a finite state space SSMarkov chain X t X_ton$S$$X_t$, if within a time step from $i$ to$The\; probability\; of\; j$ moving is$Pr(j|i)=P_{i,j}$, the random matrix $P$ 's$line\; i$ , line$The\; j$ column elements are composed of$P_{i,j}$给出：
$$P=\left[\begin{array}{cccccc}{P}_{1,1}& P_{1,2}& \dots & P_{1,j}& \dots & P_{1,S}\\ P_{2,1}& P_{2,2}& \dots & P_{2,j}& \dots & P_{2,S}\\ ⋮& ⋮& \ddots & ⋮& \ddots & ⋮\\ P_{i,1}& P_{i,2}& \dots & P_{i,j}& \dots & P_{i,S}\\ ⋮& ⋮& \ddots & ⋮& \ddots & ⋮\\ P_{S,1}& P_{S,2}& \dots & P_{S,j}& \dots & P_{S,S}\end{array}\right].$$

nature

An obvious conclusion is that the matrix $P$ has eigenvalue$1$ (corresponding to the right random matrix, where the sum of each row is 1, if the sum of each column is 1, just take the transpose verification, the same reason), that is,P
$$Px=1\cdot$$
Based on the completeness and additivity of probability, x can be verified, or in other words, since the transfer process is inevitable, the sum of each probability is 1$$.$$

When transferring from one probability space to another, the completeness, non-negativity, listability and additivity of probability still hold. Therefore if $AandB$ are two$n\times n$ 's transfer matrix, its product (probability transfer), power (self-evolution), and arithmetic mean are still transfer matrices, namely
$$AB、A^n、\frac{A+B}{2}$$

Let's think about $A^n$当$n\to$The scene at$\infty$$The\; largest\; eigenvalue\; of\; A$ is 1, and the absolute values ​​of the remaining eigenvalues ​​are less than 1, so$A^n$ will actually tend to be by$l=$The steady state represented by$1\; .$

discrete source channel model

Friends who have studied "Information Theory" should be familiar with similar forms of transfer matrices, so we use a discrete source channel model as an example. To be precise, a source can be described as a probability matrix. The so-called source distribution means that each transmitted signal occupies a certain transmission possibility. Such as row vector $P_X$To describe a ternary and equally likely source distribution, that is,
$$P_X=\left[\begin{array}{ccc}1/3& 1/3& 1/3\end{array}\right]$$
Suppose we face such a channel transfer model

, then the corresponding channel transfer matrix is
$$P_{ij}=\left[\begin{array}{cc}0.8& 0.2\\ 0.5& 0.5\\ 0.2& 0.8\end{array}\right]rowi(probability\; space\; ofX)\to Columnj(the\; probability\; space\; ofY)$$
then has a sink probability distribution of
$$P_Y=P_X{P}_{ij}=\left[\begin{array}{ccc}1/3& 1/3& 1/3\end{array}\right]\left[\begin{array}{cc}0.8& 0.2\\ 0.5& 0.5\\ 0.2& 0.8\end{array}\right]=\left[\begin{array}{cc}0.5& 0.5\end{array}\right]$$
But in fact we often face this problem, to find the capacity$C$为？
$$C=\max [H(Y)-H(Y|X)]$$
and known
$$H(Y)\le H(0.5,0.5)=1bit,H(Y|X)\ge H(0.2,0.8)=0.7219bits$$
And when the two formulas are equal, the corresponding source distribution is
$$P_X=\left[\begin{array}{ccc}1/2& 0& 1/2\end{array}\right]$$
So the channel capacity is equal to
$$C=H(0.5,0.5)-H(0.2,0.8)=0.2781bits$$
This gives people an intuitive feeling that the more the transition probability of the selected channel deviates from 0.5, the better, andthe transmission of equal probability errors is invalid.

Fourier matrix

choose a base

Fourier transform is a classic operation in signal processing. Compared with wavelet transform, the difference lies in the difference of the basis functions . So how to choose the basis function will make people feel comfortable? One answer is to choose a set of orthonormal basis . Thus, when we arrange these basis vectors together to form a transformation matrix, the matrix is ​​a normal matrix, and its inverse is its conjugate transpose .

The old man said such a sentence, "I need infinitely many, because my space is infinite dimensional." That is to say, an infinite-dimensional space should be described by an infinite-dimensional basis. Then for an n-dimensional space, we actually use n orthonormal bases, that is,
$$v=x_1{q}_1+x_2{q}_2+\cdots +x_i{q}_i+\cdots +x_n{q}_n$$
Among them $x_i$Represents the corresponding base $q_i$extension coefficient on . How to get this coefficient? This is to reflect the ingenuity of orthonormality, so that the two sides can be multiplied by the conjugate transpose vector
$$q_i^{H}v=x_1{q}_i^H{q}_1+x_2{q}_i^H{q}_2+\cdots +x_i{q}_i^H{q}_i+\cdots +x_n{q}_i^H{q}_n=x_i$$
At this time, 0 will be obtained because they are orthogonal to each other, and 1 will be obtained due to self-standardization, so that it is convenient to obtain the coefficient $x_i$。

It is worth noting that orthogonality is very useful . In signal analysis, the familiar Fourier series uses orthogonal trigonometric functions as the basis, namely $\sin (nx)$和$\cos \left(nx\right)$ , it can be verified that their integrals in a single period are 0, that is, they are orthogonal, and this idea of ​​orthogonality is also the principle of OFDM subcarrier selection.

Fourier matrix

Refer to Wikipedia for the definition of Fourier matrix.

The discrete Fourier transform of N points can be used with an $n\times The\; matrix\; multiplication\; of\; m$ is expressed, that is,$X=Fx$ , where$x$ is the original input signal,$X$ is the output signal obtained through discrete Fourier transform. a$n\times$The transformation matrix FFof$n$$F$ can be defined as$F=({oh}^{ij}{)}_{i,j=0,\dots ,N-1}/\sqrt{N}$, ie
$$F=\frac{1}{\sqrt{N}}\left[\begin{array}{cccccc}1& 1& 1& 1& \cdots & 1\\ 1& oh& {oh}^2& {oh}^3& \cdots & {oh}^{N-1}\\ 1& {oh}^2& {oh}^4& {oh}^6& \cdots & {oh}^{2(N-1)}\\ 1& {oh}^3& {oh}^6& {oh}^9& \cdots & {oh}^{3(N-1)}\\ ⋮& ⋮& ⋮& ⋮& & ⋮\\ 1& {oh}^{N-1}& {oh}^{2(N-1)}& {oh}^{3(N-1)}& \cdots & {oh}^{(N-1)(N-1)}\end{array}\right]$$

in that $w$ is defined as$1$ ofnnThe principal value of the nth root$,$ that is,
$$w=e^{\frac{-2\pi i}{N}}$$
Note that for a column $j$ , its column vector is:
$$q_j={\left[\begin{array}{c}{w}^{\alpha \times (j-1)}\end{array}\right]}^T/\sqrt{N}a=0,1,2,\cdots ,N-1$$
At this point we take another column$q_{j+k}$, make the inner product of the two, then
$$q_{j+k}^H{q}_j=\frac{1}{N}\sum _{\alpha =0}^{N-1}{w}^{\alpha \times (j-1)}{w}^{-\alpha \times (j+k-1)}=\sum _{\alpha =0}^{N-1}{w}^{-\alpha k}$$
when$k=When\; 0$ , that is to find the square of its own modulus, it can be seen that the result of the inner product at this time is 1, which verifies the standardization.
And when$-k\ne 0$ , we know that$w^{-k}$ is a$N$ times the root of unity, and$The\; coefficient\; \alpha$ acts as a rotation on the unit circle. How about$-k=1$ as an example, then it can be thought that when$\alpha$ from$0$ times to fetch$N-1$ ,this$N$ results will be reconstituted on the unit circle$1$ of$N$ NNs$N$ times the root of unity. So the question becomes, why does this$The\; sum\; of\; N$ roots of unity is$0$ ? These three explanations are given on Baidu Encyclopedia:

A basic method is the summation of geometric series , that is, $$\sum _{\alpha =0}^{\infty }{w}^a=\frac{1-w^N}{1-w}=0$$
The second proof is that they form the vertices of a regular polygon on the complex plane, andfrom the symmetry we know that the center of gravity of the polygon is at the origin. The last proof usesVedder's theorem about the roots and coefficients of the equation, from the circle-dividing equation$x_{n-1}$term but the coefficient is zero.

No matter which interpretation is adopted, the conclusion is consistent, that is, $k\ne When\; 0$ , the result of the inner product is 0, thus verifying the orthogonality. Then,the Fourier matrix is ​​a normal matrix, and an excellent property is produced, that is,
$$F^{-1}=F^H$$

FFT overhead $Nog{\_}_2$Description of$N$$/2$

First from a theoretical point of view. In digital signal processing, we know that there are two typical ideas to complete FFT, one is time selection DIT, and the other is frequency selection DIF. After a sorting, both methods will reduce the amount of calculation by half .
For DIT, for $The\; parity\; of\; the\; x\left(n\right)$ sequence is sorted, and the result is$X\left(k\right)$ is the first half and the second half.


ForDIF, it is for$The\; first\; half\; and\; the\; second\; half\; of\; the\; x\left(n\right)$ sequence are sorted, and the result is output$X\left(k\right)$ is divided into odd and even.

It is also known that the definition of DFT operation is
$$X(k)=\sum _{n=0}^{N-1}x(n)w_N^{nk},k=0,1,\cdots ,N-1$$

Among operations, multiplication has the largest operational overhead . for $For\; the\; DFT\; of\; N$ points, the cost of matrix multiplication is N, there are N points in total, and the cost is estimated to be$N^2$ , and FFT makes the final conversion into$lo{g}_{2}For\; N$ -level butterfly operations, the multiplication cost of butterfly operations at each stage is$N/2$ (that is, every two coefficients can be shared once, saving half of the calculation), so the total cost is$Nog{\_}_{2}N/2$。

Now we understand the above process from the perspective of matrix transformation, let's show the process of converting a 64-point FFT to a 32-point FFT:
$$F_{64}=\left[\begin{array}{cc}I& D_{32}\\ I& -D_{32}\end{array}\right]\left[\begin{array}{cc}{F}_{32}& 0\\ 0& F_{32}\end{array}\right]\left[\begin{array}{ccccccc}1& 0& 0& 0& \cdots & 0& 0\\ 0& 0& 1& 0& \cdots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \cdots & 1& 0\\ 0& 1& 0& 0& \cdots & 0& 0\\ 0& 0& 0& 1& \cdots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \cdots & 0& 1\end{array}\right]odd\; rowseven\; rows$$
where $D_{32}$is a diagonal matrix for correction, and the elements are the familiar correction factors $w$ , we can find that it is the correction coefficient in the butterfly operation, that is,
$$D_{32}=\left[\begin{array}{ccccc}{w}^0& 0& 0& 0& \cdots \\ 0& w^1& 0& 0& \cdots \\ 0& 0& w^2& 0& \cdots \\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& 0& w^{31}\end{array}\right]$$
The rightmost matrix is ​​the switching matrix $P_{64}$, to realize the function of parity sorting . Observing this process, we found that the increased multiplication overhead is mainly in $D_{32}$, so the computation of this stage is transformed into two $F_{32}$The amount of multiplication and a $D_{32}$of multiplication operations. And so on, $F_{32}$It can also be further decomposed into $F_{16}\cdots$ Then the resulting block of basic elements will be$F_2$, shown as follows
$$\left[\begin{array}{cc}I& D_{32}\\ I& -D_{32}\end{array}\right]\left[\begin{array}{cc}\begin{array}{cc}I& D_{16}\\ I& D_{16}\end{array}& \text{0}\\ \text{0}& \begin{array}{cc}I& D_{16}\\ I& -D_{16}\end{array}\end{array}\right]\cdots \left[\begin{array}{cc}\begin{array}{ccc}I& D_1& \\ I& D_1& \\ & & \ddots \end{array}& \text{0}\\ \text{0}& \begin{array}{cc}I& D_1\\ I& -D_1\end{array}\end{array}\right]\left[\begin{array}{cc}\begin{array}{cc}{F}_2& \\ & \ddots \end{array}& \text{0}\\ \text{0}& \begin{array}{c}{F}_2\end{array}\end{array}\right]\left[\begin{array}{cc}\begin{array}{cc}{P}_2& \\ & \ddots \end{array}& \text{0}\\ \text{0}& \begin{array}{c}{P}_2\end{array}\end{array}\right]\cdots \left[\begin{array}{cc}{P}_{32}& 0\\ 0& P_{32}\end{array}\right]\left[\begin{array}{c}{P}_{64}\end{array}\right]$$
It can be seen thatthe correction matrix has a total of${\log }_{2}N$ levels, each level has$N/2$ multiplication operations. Therefore, the final operation cost is$Nog{\_}_{2}N/2$ . Let's intuitively feel the reduction of the amount of calculation, with$N=Take\; 1024$ as an example,
$$\frac{N^2}{Nog{\_}_{2}N/2}=\frac{1024^2}{1024\ast 5}\approx 205$$
It can be seen that FFTprovides the possibility for high performance of Fourier transform processing.

Explanation of Diagonalization of Circular Matrix

The background of this description is the cyclic prefix in OFDM, refer to Wikipedia, first look at the definition

形式为$$A=\left[\begin{array}{ccccc}{c}_0& c_{n-1}& c_{n-2}& \dots & c_1\\ c_1& c_0& c_{n-1}& \cdots & c_2\\ c_2& c_1& c_0& \cdots & c_3\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ c_{n-1}& c_{n-2}& c_{n-3}& \dots & c_0\end{array}\right]$$The matrix of is a circulant matrix.

We focus on this property: the eigenvector matrix of the circulatory matrix is ​​a discrete Fourier transform matrix of the same dimension, so the eigenvalues ​​of the circulatory matrix can be easily calculated by fast Fourier transform, or the circulatory matrix can be calculated by the Fourier matrix Diagonalization , that is:
$$AF=F\Lambda \Rightarrow L=F^{H}AForA=F\Lambda F^H$$
Let us briefly prove that there are three main preconditions: one is the definition of DFT, the other is its time-domain cyclic shift property, and the third is symmetry. That is,
$$\begin{gathered} Condition1:X\left(k\right)=\sum _{n=0}^{N-1}x(n)w_N^{nk},k=0,1,\cdots ,N-1 \\ Condition2:x((n+m){)}_N{R}_N(n)\iff w_N^{-km}X(k) \\ Condition3:x(N-n)\iff X(N-k) \end{gathered}$$
Take the second row as an example, we can see that
$$c_1{w}^{0\times 1}+c_0{w}^{1\times 1}+c_{N-1}{w}^{2\times 1}+\cdots +c_2{w}^{N-1\times 1}=\sum _{n=0}^{N-1}c((N-n+1){)}_N{R}_N(n)w^{n\times 1}=w^{-1\times 1}\sum _{n=0}^{N-1}c((N-n){)}_N{R}_N\left(n\right)w^{n\times 1}=C(N-1)w^{-1\times 1}whenk=1$$
and so on, we can know$i$ is the shifted$m$，$i$ is also kkin the DFT result$k$，$j$ is equivalent to serial number$n$ 。
$$\sum _{j=0}^{N-1}c((N-j+i){)}_N{R}_N(n)w^{j\times i}=C((N-i){)}_N{R}_N\left(n\right)w^{-i^2}i(k,m),j\left(n\right)=0,1,2,\cdots ,N-1$$
Write the above form as a matrix, namely
$$\left[\begin{array}{c}c((N-j+i){)}_N{R}_N(n)\end{array}\right]\left[\begin{array}{c}{w}^{i\times j}\end{array}\right]=\left[\begin{array}{c}C((N-i\left){)}_N{R}_N\right(n)\end{array}\right]\left[\begin{array}{c}{w}^{-i^2}\end{array}\right]$$
on both sides and divide$\sqrt{N}$Normalized, it can be written in the form of Fourier matrix
$$AF=\left[\begin{array}{ccccc}C(0)& 0& 0& \cdots & 0\\ 0& C(N-1)& 0& \cdots & 0\\ 0& 0& C(N-2)& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 0& 0& 0& 0& C(1)\end{array}\right]F^H=\Lambda F^H=H{F}^H$$
From this proof process, we can also see thatthe$Line\; i$ input signal sequence$c((N-j+i){)}_N{R}_N(n)$ , the eigenvalue of the corresponding output is$C((N-i\left){)}_N{R}_N\right(n)$ . The purpose of this is toconvert the convolution operation corresponding to the channel response into an FFT operation, and the FFT has high-performance operation efficiency, or in other words, the eigenvalue at this time corresponds to the channel response. It can be seen that the advancement of digital signal processing technology has laid the foundation for the advancement of communication technology, and all of this requires mathematics.

Note that if the circulant matrix is ​​written in the following form
$$A=\left[\begin{array}{ccccc}{c}_0& c_1& c_2& \dots & c_{n-1}\\ c_{n-1}& c_0& c_1& \cdots & c_{n-2}\\ c_{n-2}& c_{n-1}& c_0& \cdots & c_{n-3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ c_1& c_2& c_3& \dots & c_0\end{array}\right]$$
Then the eigenvalue obtained is $C\left(i\right)$ , which is also the example matrix given in the book "Toeplitz and Circulant Matrices: A review". The book gives a general proof of the diagonalization of circulant matrices (Chapter 3), the difference is that I talk about it more simply by combining the properties of DFT here.