Methods for model structure optimization: standardization, normalization, random discarding

Standardization, Normalization and Dropout are some techniques commonly used in neural network training. They have different functions and uses:

1. Standardization :

   • What it does : Standardization is a method of preprocessing data so that it has zero mean and unit variance before being input to the model. This can help speed up the training process, allowing optimization algorithms such as gradient descent to converge more quickly and stably.

   • Formula : Assuming (x) is a feature, its standardized value (x_{\text{std}}) is calculated as follows: [x_{\text{std}}
     = \frac{x - \mu}{\sigma}]
     Among them, (\mu) is the sample mean and (\sigma) is the sample standard deviation.

   • Implementation : In practice, normalization can be done using library functions or manual calculations.

2. Normalization :

   • Function : Normalization is also a method of preprocessing data, scaling the data to a specific range, usually [0, 1] or [-1, 1]. It helps ensure that all features contribute to model training to a similar degree and avoid optimization problems caused by excessive differences in feature values.

   • Formula : The calculation formula of normalization varies depending on the specific method. The most common is Min-Max normalization:
     [x_{\text{norm}} = \frac{x - \min(x)}{\max( x) - \min(x)}]
     where (\min(x)) is the minimum value of the sample, and (\max(x)) is the maximum value of the sample.

   • Implementation : Normalization can also be performed using library functions or manual calculations.

3. Dropout :

   • Role : Dropout is a regularization technique that can be used to reduce overfitting in neural networks. It randomly sets some neurons to 0 with a certain probability during the training process, thereby reducing the dependence between neurons and making the model more robust and capable of generalization.

   • Implementation : During training, neurons in the hidden layer are randomly set to zero with a certain probability (usually between 0.2 and 0.5). During testing, no discarding is performed, but the output values ​​of neurons discarded during training need to be scaled according to a certain proportion to keep the expected output consistent.

   • Note : Dropout is only used during training, not during testing.

These technologies are important means to improve the performance, generalization ability and stability of neural network models. In practical applications, appropriate preprocessing and regularization techniques are usually selected based on the characteristics of specific tasks and data.