Source: https://blog.csdn.net/sinat_32043495/article/details/78841553
- Full connection layer / FC layer
All the neurons between layers are connected with weights
typically fully connected layers tail convolutional neural network
- Excitation layer (RELU) the convolution output layer as nonlinear mapping
the Sigmoid
Tanh (hyperbolic tangent)
RELU
the Leaky RELU
the ELU
MAXOUT
gradient disappears: This is essentially due to the activation function selection result, the most simple sigmoid function, for example, seeking gradient function at both ends of the guide result is very small (saturation region), resulting in the activation due to multiple function derivative value is used such that the overall result of the product of the gradient becomes smaller propagation process, also appeared gradient the disappearance of the phenomenon.
Gradient explosion: Similarly, appears in the activation function in the active region, but under the weight W is too large. But not as good as the explosion gradient gradient disappeared and more opportunities arise.