Learning Algorithm Summary

Target

\hat{w} = a r g m i n_{w} \sum_{i = 1}^{n} L (w, z_{i}), | | w | |_{1} \leq s

$\widehat{w}=argmin_w\sum_{i=1}^{n}L(w, z_i), ||w||_1 \le s$

\hat{w} = a r g m i n_{w} \sum_{i = 1}^{n} L (w, z_{i}) + λ | | w | |_{1}

$\widehat{w}=argmin_w\sum_{i=1}^{n}L(w, z_i) + \lambda||w||_1$

Gradient Descent

W^{(t + 1)} = W^{(t)} - η^{(t)} G^{(t)} = W^{(t)} - η^{(t)} \nabla ζ_{W} (W^{(t)}, Z)

$W^{(t+1)}=W^{(t)}-\eta^{(t)}G^{(t)}=W^{(t)}-\eta^{(t)}\nabla \zeta_W(W^{(t)}, Z)$

Stochastic Gradient Descent

W^{(t + 1)} = W^{(t)} - η^{(t)} G_{j}^{(t)} = W^{(t)} - η^{(t)} \nabla ζ_{W} (W^{(t)}, Z_{j})

$W^{(t+1)}=W^{(t)}-\eta^{(t)}G_j^{(t)}=W^{(t)}-\eta^{(t)}\nabla \zeta_W(W^{(t)}, Z_j)$

Momentum

m_{t} = μ m_{t - 1} + G^{(t)}

$m_t = \mu m_{t-1}+G^{(t)}$

W^{(t + 1)} = W^{(t)} - η^{(t)} m_{t}

$W^{(t+1)} = W^{(t)} - \eta^{(t)}m_t$

Nesterov

扫描二维码关注公众号，回复： 2332826 查看本文章

m_{t} = μ m_{t - 1} + G^{(t)}

$m_t = \mu m_{t-1}+G^{(t)}$

W^{(t + 1)} = W^{(t)} - η μ m_{t - 1} - η G^{(t)}

$W^{(t+1)}=W^{(t)}-\eta \mu m_{t-1} - \eta G^{(t)}$

Adagrad

n_{t} = n_{t - 1} + (G^{(t)})^{2}

$n_t=n_{t-1}+(G^{(t)})^2$

W^{(t + 1)} = W^{(t)} - \frac{η}{\sqrt{n_{t} + ϵ}} G^{(t)}

$W^{(t+1)}=W^{(t)}-\frac{\eta}{\sqrt{n_t+\epsilon}}G^{(t)}$

Adadelta

n_{t} = ν n_{t - 1} + (1 - ν) (G^{(t)})^{2}

$n_t=\nu n_{t-1}+(1-\nu)(G^{(t)})^{2}$

W^{(t + 1)} = W^{(t)} - \frac{η}{\sqrt{n_{t} + ϵ}} G^{(t)}

$W^{(t+1)}=W^{(t)}-\frac{\eta}{\sqrt{n_t+\epsilon}}G^{(t)}$

With L1 Regulization

W^{(t + 1)} = W^{(t)} - η^{(t)} G^{(t)} - η^{(t)} λ s g n (W^{(t)})

$W^{(t+1)}=W^{(t)}-\eta^{(t)}G^{(t)}-\eta^{(t)}\lambda sgn(W^{(t)})$

Simple Truncated

T_{0} (v, θ) = {\begin{aligned} 0 i f | v | \leq θ \\ v o t h e r w i s e \end{aligned}

$T_0(v, \theta)=\left\{ \begin{aligned} 0 \space\space if |v| \le \theta \\ v \space\space otherwise \\ \end{aligned} \right.$

W^{(t + 1)} = T_{0} (W^{(t)} - η^{(t)} G^{(t)}, θ)

$W^{(t+1)}=T_0(W^{(t)}-\eta^{(t)}G^{(t)}, \theta)$

Truncated Gradient

T_{1} (v, α, θ) = {\begin{aligned} m a x (0, v - α) & i f v \in [0, θ] \\ m i n (0, v + α) & i f v \in [- θ, 0] \\ v & o t h e r w i s e \end{aligned}

$T_1(v, \alpha, \theta)=\left\{ \begin{aligned} max(0, v-\alpha) & &if\space v \in [0, \theta] \\ min(0, v+\alpha) & &if\space v \in [-\theta, 0] \\ v & & otherwise \end{aligned} \right.$

W^{(t + 1)} = T_{1} (W^{(t)} - η^{(t) G^{(t)}}, η^{(t)} λ^{(t)}, θ)

$W^{(t+1)}=T_1(W^{(t)}-\eta^{(t)G^{(t)}}, \eta^{(t)}\lambda^{(t)}, \theta)$

Adam

m_{t} = μ m_{t - 1} + (1 - μ) G^{(t)}

$m_t=\mu m_{t-1}+(1-\mu)G^{(t)}$

n_{t} = ν n_{t - 1} + (1 - ν) (G^{(t)})^{2}

$n_t=\nu n_{t-1} + (1-\nu)(G^{(t)})^2$

{\hat{m}}_{t} = \frac{m_{t}}{1 - μ^{t}}

$\widehat m_t=\frac{m_t}{1-\mu^{t}}$

{\hat{n}}_{t} = \frac{n_{t}}{1 - ν^{t}}

$\widehat n_t = \frac{n_t}{1-\nu^{t}}$

W^{(t + 1)} = W^{(t)} - \frac{{\hat{m}}_{t}}{\sqrt{{\hat{n}}_{t}} + ϵ} η

$W^{(t+1)}=W^{(t)}-\frac{\widehat m_t}{\sqrt{\widehat n_t} + \epsilon}\eta$

FOBOS

W^{(t + 0.5)} = W^{(t)} - η^{(t)} G^{(t)}

$W^{(t+0.5)}=W^{(t)}-\eta^{(t)}G^{(t)}$

W^{(t + 1)} = a r g m i n_{w} {\frac{1}{2} | | W - W^{(t + 0.5)} | |_{2}^{2} + η^{(t + 0.5)} Ψ (W)}

$W^{(t+1)}=argmin_w\{\frac{1}{2}||W-W^{(t+0.5)}||_2^2+\eta^{(t+0.5)}\Psi(W) \}$

RDA

W^{(t + 1)} = a r g m i n_{w} {\frac{1}{t} \sum_{r = 1} t G^{(r)} \cdot W + Ψ (W) + \frac{β^{(t)}}{t} h (w)}

$W^{(t+1)} = argmin_w\{\frac{1}{t}\sum_{r=1}{t}G^{(r)} \cdot W + \Psi(W) + \frac{\beta^{(t)}}{t}h(w)\}$

FTRL

W^{(t + 1)} = a r g m i n_{w} {G^{(1 : t)} \cdot W + λ_{1} | | W | |_{1} + \frac{λ_{2}}{2} + \frac{1}{2} \sum_{s = 1}^{t} | | W - W^{(s)} | |_{2}^{2}}

$W^{(t+1)}=argmin_w\{G^{(1:t)}\cdot W + \lambda_1||W||_1+\frac{\lambda_2}{2}+\frac{1}{2}\sum_{s=1}^{t}||W-W^{(s)}||_2^2\}$