条件熵
\[ \begin{equation} H(Y | X) \quad=\sum_{x \in X} p(x) H(Y | X=x) \end{equation} \]
信息增益：熵 - 条件熵
\[ \begin{aligned} i_{t} &=\sigma\left(W_{i i} x_{t}+b_{i i}+W_{h i} h_{(t-1)}+b_{h i}\right) \\ f_{t} &=\sigma\left(W_{i f} x_{t}+b_{i f}+W_{h f} h_{(t-1)}+b_{h f}\right) \\ g_{t} &=\tanh \left(W_{i g} x_{t}+b_{i g}+W_{h g} h_{(t-1)}+b_{h g}\right) \\ o_{t} &=\sigma\left(W_{i o} x_{t}+b_{i o}+W_{h o} h_{(t-1)}+b_{h o}\right) \\ c_{t} &=f_{t} * c_{(t-1)}+i_{t} * g_{t} \\ h_{t} &=o_{t} * \tanh \left(c_{t}\right) \end{aligned} \]
到底是拼接起来，还是两个向量分别由一个权重矩阵处理后相加起来？