Adaptive filtering method - LMS algorithm

adaptive filter

• Adaptive filter: a digital filter that can automatically adjust its parameters according to the input signal

• Non-adaptive filter: a digital filter with a static filter system, the static coefficients forming the transfer function of the filter

• For some applications (such as system identification, prediction, denoising, etc.), the parameters to be operated cannot be known in advance, and adaptive coefficients must be used for processing. In this case, adaptive filters are usually used

• When an adaptive filter processes a speech signal, it does not need to know the statistical characteristics of the input signal and noise. The filter itself can learn or estimate the statistical characteristics of the signal during the working process, and adjust its own parameters based on this to achieve a certain Optimal filtering effect under criterion/cost function

• Adaptive filter is an effective means to deal with non-stationary signals
  

• Let's start with an N-order filter and gradually transition to an adaptive filter

N order filter

• An N-order filter with parameters $\text{w}\left(n\right)$ , contains N independent filter coefficients
• The output of the filter is a linear convolution of the input signal and the filter coefficients:
  $$and\left(n\right)=\sum _{i=0}^{N-1}{w}_i(n)x(n-i)={\text{w}}^T(n)\text{x}(n)={\text{x}}^T(n)\text{w}(n)$$
• in,
  • $\text{x}(n)=[x(n),x(n-1),...,x(n-N+1){]}^T$
  • $\text{w}(n)=[w_0(n),w_1(n),...,w_{N-1}(n){]}^T$
  • n in parentheses represent different moments
• Define the desired output as d(n), and the error sequence is: $e(n)=d(n)-and\left(n\right)=d\left(n\right)-{\text{w}}^T(n)\text{x}(n)$
• Since the error at time n is defined in terms of the filter coefficients at time n, this error is called "a posteriori error"

Standard LMS algorithm

• According to the minimum mean square error (MMSE) criterion, the objective function to minimize is
  $$J(\text{w})=E[|e(n){|}^2]=E[|d(n)-{\text{w}}^T(n)\text{x}(n){|}^2]$$
• In order to minimize the mean square error function, it is necessary to calculate $J\left(\text{w}\right)$ VS w's derivation, derivation is zero:
  $$E[\text{x}(n){\text{x}}^T(n)]\text{w}(n)-E[\text{x}(n)d(n)]=R\text{w}-r=0$$
• Can get: ${\text{w}}_{opt}=R^{-1}r$
• 称${\text{w}}_{opt}$The defined filter is a Wiener filter, and the Wiener filter is a statistically optimal filter in the sense of the smallest mean square error
• Disadvantages of Wiener filter:
  • Not suitable for non-stationary processes, and speech is a non-stationary signal
  • The calculation of mathematical expectation needs to use all historical data to estimate the current filter coefficient, which is not a reasonable choice
• Due to the mathematical expectations involved, in practical applications, the true autocorrelation matrix R of the signal and the cross-correlation vector r between the input signal and the expected output cannot be obtained. Therefore, in practical applications, the square of the instantaneous error is used to replace the mathematical expectation in the objective function, and then the instantaneous gradient is obtained: $▽(n)=-2e\left(n\right)\text{x}\left(n\right)$ , and then update the filter coefficients along the negative gradient direction:
  $$\text{w}(n+1)=\text{w}(n)+2\mu \text{x}(n)e(n)$$
• The above formula is the update formula of the standard time-series LMS algorithm, which is updated point by point. Whenever a new x(n) and d(n) are given, the filter coefficients are updated once
• The execution flow of the standard LMS algorithm:
  • Initialize w(0), x(0)
  • For each new input value x(n), calculate the output value y(n)
  • Using the expected output d(n), calculate the error value e(n) to get the gradient
  • Update the filter coefficients to get w(n+1)
  • Return to step 2 until the end, the output sequence and error sequence can be obtained
• Standard LMS algorithm
  • The basic idea is gradient descent
  • The advantage is that the algorithm is simple and easy to implement
  • The disadvantage is that the convergence speed is slow, and the tracking performance is relatively poor due to the presence of gradient noise

block (block) LMS algorithm

• With more sample points, update the filter coefficient once:
  $$\begin{array}{rl}\text{w}(n+1)& =\text{w}(n)+2\mu \text{x}(n)e(n)\\ \text{w}(n+2)& =\text{w}(n+1)+2\mu \text{x}(n+1)e(n+1)\\ ...& \\ \text{w}(n+L)& =\text{w}(n+L-1)+2\mu \text{x}(n+L-1)e(n+L-1)\end{array}$$
• 得：$\text{w}(n+L)=\text{w}(n)+2m{\sum }_{m=0}^{L-1}\text{x}(n+m)e(n+m)$
• Update in blocks, update the filter coefficients every L points
• Note that the error sequence is obtained through the same filter: $e(n+m)=d(n+m)-{\text{x}}^T(n+m)\text{w}(n)$
• For the convenience of subsequent pushes, let L=N, that is to say, when the number of collected sampling points is equal to the order of the filter, the filter coefficients are updated.
• Compared with the standard LMS, the update frequency of the block LMS is reduced by L times, so a new time index k is defined to replace n, $k=\frac{n}{L}$
• Thus the block LMS update formula is: $\text{w}(k+1)=\text{w}(k)+2m{\sum }_{m=0}^{L-1}\text{x}(kL+m)e(kL+m)$
• In block LMS, the instantaneous gradient is: $▽(k)=-2{\sum }_{m=0}^{L-1}\text{x}(kL+m)e(kL+m)$
• The core operation of block LMS:
  • The output sequence is the linear convolution of the input signal and the filter coefficients: $and(n)+m)={\text{x}}^T(n+m)\text{w}(n)，m\in [0,N-1]$
  • The instantaneous gradient is the linear correlation between the input signal and the error sequence: $▽(k)=-2{\sum }_{m=0}^{L-1}\text{x}(kL+m)e(kL+m)$
• There are Overlap-save method and Overlap-add method for calculating linear convolution using FFT

Overlap-save method

• The relationship between linear convolution and circular convolution: Generally, if the lengths of two finite length sequences are $N_1、N_2$, and satisfy $N_1\ge N_2$, then the post N 1 − N 2 + 1 N_1-N_2+1 of the circular convolution$N_1-N_2+1$ point, consistent with the result of linear convolution

• The relationship between linear correlation and circular correlation: Generally, if the lengths of two finite length sequences are $N_1、N_2$, and satisfy $N_1\ge N_2$, then the first N 1 − N 2 + 1 N_1-N_2+1 of the circular correlation$N_1-N_2+1$ point, consistent with results for linear correlation

• If you want to understand the above two relationships, you can refer to the basic operations of digital signals - linear convolution (correlation) and circular convolution (correlation)

• In order to get the output sequence of N points: $and(n)+m)={\text{x}}^T(n+m)\text{w}(n)，m\in [0,N-1]$ , it is necessary to ensure that the results of linear convolution and circular convolution of at least N points are the same, that is,$N_1-N_2+1\ge N$

• Because $N_1\ge N_2$, and the filter order is N, so $N_2=N$ , so the condition becomes:$N_1\ge 2N\_-1$

• In order to facilitate the calculation of FFT, let $N_1=2N\_$

• According to the figure below, construct x(n) and w(n)
  

• Calculate the Fourier transform of the input signal vector and the filter vector separately
  X ( k ) = diag { F [ x ( k N − N ) , . . . , x ( k N − 1 ) , x ( k N ) , . . . , x ( k N + N − 1 ) ] } W ( k ) = F [ w T ( k ) , 0 , . . . , 0 ] TY ( k ) = X ( k ) W ( k ) \begin {aligned} X(k)&=diag\{ F[x(kN-N), ..., x(kN-1), x(kN), ..., x(kN+N-1)] \} \\ W(k)&=F[\text{w}^T(k), 0, ..., 0]^T \\ Y(k)&=X(k)W(k) \ end{aligned}X(k)W(k)Y ( k )=diag{ F[x(kN−N),...,x(kN−1),x(kN),...,x(kN+N−1)]}=F[wT(k),0,...,0]T=X(k)W(k)​

• The length of each Y(k) is 2N, and the N-point linear convolution of the output sequence y(k) is equal to the N points of Y(k) after Fourier inverse transformation y ( k ) = [ y ( k N
  $$\begin{array}{rl}and\left(k\right)& =\left[y\right(kN),y(kN+1),y(kN),...,y(kN+N-1)]\\ & =\text{last N samples of }{F}^{-1}Y(k)\end{array}$$

• The calculation process of y(k) is as follows:
  

• Next calculate the sum of the instantaneous gradients of N points: $▽(k)=-2{\sum }_{m=0}^{L-1}\text{x}(kL+m)e(kL+m)$

• For linear correlation operations, it is still transformed to the frequency domain, and the product of the conjugate spectrum of the input signal and the error sequence is calculated

• X(k) has been obtained above, and then the error sequence $e(k)=\left[e\right(kN),e(kN+1),...,e(kN+N-1)]$ is also extended to a length of 2N

• Note: When calculating convolution, add N 0s after w(n) , and when calculating correlation, add N 0s in front of e(n)

• Fourier transform of the error sequence $E(k)=F[0,0,...,0,e^T(k){]}^T$

• Fourier transform of instantaneous gradient $▽(k)=X^H(k)E(k)$

• For each $The\; length\; of\; ▽\left(k\right)$ is 2N, and the N-point linear correlation of the sum of instantaneous gradients is equal to$▽\left(k\right)$ the first N points after inverse Fourier transform
  $$\begin{array}{rl}\overrightarrow{▽}(k)& =[▽(kN),▽(kN+1),▽(kN),...,▽(kN+N-1)]\\ & =\text{first N samples of }{F}^{-1}▽(k)\end{array}$$

Frequency-Domain Adaptive Filter（FDAF）

• The update of the filter coefficients can also be done in the frequency domain $W(k+1)=W(k)+2\mu F[{\overrightarrow{▽}}^T(k),0,...,0{]}^T$
• Notice
  • The filter coefficients are updated directly in the frequency domain, so the gradient vector needs to be transformed to the frequency domain again
  • Since the filter coefficients are filled with N zeros in the back, the gradient vector also needs to be filled with N zeros in the back.
• The overall framework of FDAF is as follows: