initialization
The essence of the deep learning model training process is to update the parameter w, which requires each parameter to have a corresponding initial value.
Why initialize?
The neural network needs to optimize a very complex nonlinear model, and there is basically no global optimal solution, and initialization plays a very important role in it.
□ The selection of the initial point can sometimes determine whether the algorithm converges;
□ When it converges, the initial point can determine how fast the learning converges and whether it converges to a point with high or low cost; □
Too large initialization leads to gradient explosion, too small The initialization of leads to vanishing gradients.
What is a good initialization?
A good initialization should meet the following two conditions:
□ The activation value of each layer of neurons will not be saturated;
□ The activation value of each layer cannot be 0.
All zero initialization: The parameters are initialized to 0.
Disadvantages: Neurons in the same layer will learn the same features, and the symmetry properties of different neurons cannot be broken.
If the weights of the neurons are initialized to 0, the output of all neurons will be the same, except for the output, all the nodes of the intermediate layer will have the value of zero. The general neural network has a symmetrical structure, so the updated network parameters will be the same during the first error backpropagation, and the same network parameters will not be able to extract useful features in the next update, so the deep learning model Neither will initialize all parameters with 0.
parameter adjustment
Batch batchsize chooses an exponential multiple of 2 to match the computer memory
Hyperparameter tuning method
Trial and error, web search, random search, Bayesian optimization, Gaussian process
