ESN 收缩分析

状态方程：

$s_t = (1-\alpha) s_{t-1} + \alpha \tanh (As_{t-1} + By_{t-1})$

或者

$\begin{array}{ll} s_{t+1} &= (1-\alpha) s_{t} + \alpha \tanh (As_{t} + B y_{t}) \\ &\triangleq f(s_t, y_t) \end{array}$
$\begin{array}{ll} \frac{\partial f_t}{\partial s_t} &= (1-\alpha) I + \alpha \frac{\partial}{\partial s_t} \left [ \begin{array}{c} \tanh(\sum_iA_{1i}s_{t,i} + \sum_iB_{1i}y_{t,i}) \\ \tanh(\sum_iA_{2i}s_{t,i} + \sum_iB_{1i}y_{t,i}) \\ \vdots \\ \tanh(\sum_iA_{ni}s_{t,i} + \sum_iB_{1i}y_{t,i}) \\ \end{array} \right] \\\\ &= (1-\alpha) I + \alpha \left [ \begin{array}{c} [1-\tanh^2(\sum_iA_{1i}s_{t,i}+ \sum_iB_{1i}y_{t,i})]A_{11} & \cdots & [1-\tanh^2(\sum_iA_{1i}s_{t,i}+ \sum_iB_{1i}y_{t,i})]A_{1n}\\ \vdots & \ddots &\vdots\\ [1-\tanh^2(\sum_iA_{ni}s_{t,i}+ \sum_iB_{1i}y_{t,i})]A_{n1} & \cdots & [1-\tanh^2(\sum_iA_{ni}s_{t,i}+ \sum_iB_{1i}y_{t,i})]A_{nn} \end{array} \right] \\\\ &= (1-\alpha) I + \alpha [I - diag(\tanh^2(As_t + By_t)) ]A \end{array}$
所以对于谱范数
$\begin{array}{ll} \left|\left|\frac{\partial f_t}{\partial s_t}\right|\right|_2 & \leq (1-\alpha) + \alpha \left|\left| I - diag(\tanh^2(As_t + By_t)) \right|\right|_2 \cdot \left|\left| A\right|\right|_2 \\ &\leq 1-\alpha + \alpha ||A||_2 \\ &\leq 1 \end{array}$