Wu Enda Improves Deep Neural Network Chapter Notes (3) - Parameter Debugging and Batch Normalize

Video course link:
https://www.bilibili.com/video/BV1FT4y1E74V?
Note reference link:
https://blog.csdn.net/weixin_36815313/article/details/105728919

1. Debugging Process (Tuning Process)

Changes to a neural network involve setting many different hyperparameters. The priorities for hyperparameter tuning are as follows:

• first priority
  • Learning rate $a$
• second priority
  • Momentum parameter $\beta$ (generally 0.9)
  • The size of the mini-batch
  • number of hidden units
• third priority
  • The number of layers of the neural network
  • Learning rate decay factor
• fourth priority
  • Adam algorithm parameter $\beta 1$ (generally 0.9)
  • Adam algorithm parameter $\beta 2$ (generally 0.999)
  • Bias term $ϵ$ (generally take$10^{-8}$）

In the earlier generation of machine learning algorithms, if you had two hyperparameters, called hyperparameter 1 and hyperparameter 2, the common practice was to sample points in the grid and then systematically study these values ​​(as shown above) .

In the field of deep learning, our common method is to randomly select points (as shown above), you can select the same number of points, and then use these randomly selected points to test the effect of hyperparameters. The reason for this is that it is difficult to know in advance which hyperparameters are most important for the problem you are trying to solve.

In fact, you may search for more than two hyperparameters. If you have three hyperparameters, then what you search is not a square, but a cube (as shown above), the hyperparameter 3 represents the third dimension, and then take the value in the three-dimensional cube, you will try more values.

Another convention is to use a coarse-to-fine strategy when you value hyperparameters .
Take two-dimensional hyperparameters as an example, take the value of hyperparameters, maybe you will find a certain point with the best effect, maybe some other points around this point are also good, then the next thing to do is to zoom in This small area (small blue box), and then get more intensive or random values ​​in it, gather more resources, and search in this blue box. If you doubt the optimal results of these hyperparameters in this area, after a rough search in the whole square, you will know that you should focus on smaller squares next. In smaller squares, you can take points more densely, so this kind of coarse-to-fine search is also often used.
By experimenting with different values ​​of hyperparameters, you can choose the optimal value for the training set objective, or the optimal value for the development set, or what you most want to optimize during the hyperparameter search process.

2. Select the appropriate range for the hyperparameters (Using an Appropriate Scale to Pick Hyperparameters)

In the range of hyperparameters, random values ​​can improve your search efficiency, but random values ​​are not random uniform values ​​within the effective range, but to choose appropriate scales for exploring these hyperparameters.

2.1 Linear axis scaling

Suppose you want to select the number of hidden units $n^{\left[l\right]}$ , the range of hyperparameters is$[50,100]$ . In this case, making a number line from 50-100 with random points on it is a more intuitive way to search for specific hyperparameters.

If you want to choose the number of layers of the neural network, assuming that the number of layers is a value from 2 to 4, so you can randomly sample uniformly in the order 2, 3, 4, or apply grid search, you will feel that 2, 3, 4 These three values ​​are reasonable.
Here are a few examples of random uniform values ​​in the range you consider, which are reasonable, but inappropriate for some hyperparameters.

2.2 Log axis scale

Suppose you are searching for a hyperparameter $\alpha$ (learning rate), its value range may be in$[0.0001,1]$内。如果画一条从0.0001到1的数轴，沿其随机均匀取值，那90%的数值将会落在0.1到1之间，即在0.1到1之间，应用了90%的资源，而在0.0001到0.1之间，只有10%的搜索资源。
因此这里使用对数标尺搜索超参数会更合理，而不使用线性轴，分别依次取0.0001，0.001，0.01，0.1，1，在对数轴上均匀随机取点，这样，在0.0001到0.001之间，就会有更多的搜索资源可用。
对数标尺转化在Python中的实现方法：

r = -4 * np.random.rand()
a = np.power(10,r)

np.random.rand()的作用是返回一个取值范围在$[0,1)$内的随机样本值，从而可以得到$r\in [-4,0)$，那么$a\in [10^{-4},10^0)$，即$a\in [0.0001,1)$ .
If you are at$10^a$ and$10^b$ , in this example,$10^a=0.0001$ , so you can pass$a=lg0.0001$ calculate$The\; value\; of\; a$ , which is -4, can be calculated in the same way asbbThe value of$b$$b=lg1$ , which is 0. All you have to do is in$[a,b]$ The interval is random and evenly given to$r$ value, then set$the\; value\; of\; a$ .

Another example is to give $The\; value\; of\; \beta$ is used to calculate the weighted average of the index. Suppose$The\; value\; range\; of\; \beta$ is$\left[\mathrm{0.9,0.999}\right].\_\_\_\_\_\_\_$ _ When computing the weighted average of the indices,$A\; beta$ of 0.9 is equivalent to calculating the average among 10 values, while a value of 0.999 is equivalent to averaging among 1000 values. So what we want to explore is$1-b\text{1−}1-\beta$ , the range of this value is$[0.1,0.001]$ , so usinga logarithmic scale, 0.1 corresponds to$10^{-1}$ , 0.001 corresponds to$10^{-3}$ , so all you have to do is in$[-3,-1]$ randomly and uniformly give$The\; value\; of\; r$ , set$1-b=10^r$，即$b=1-10^r$ , which becomes a random value of hyperparameters within a specific selection range. In this way to get the desired result, you spend as much resources searching for hyperparameters between 0.9 and 0.99 as you do searching between 0.99 and 0.999.
As for why the linear axis is not used, it is becausewhen$bAs\; \beta$ approaches 1, the sensitivity of the obtained results changes, even if$b\beta$ has slight variations. Suppose$\beta$ takes a value between 0.9 and 0.9005, according to the formula$\frac{1}{1-\beta }$可知，在这个范围内取值就相当于大概10个值取平均，因此你的结果几乎不会变化。而$\beta$值如果在0.999到0.9995之间，这会对你的算法产生巨大影响。当$\beta$取0.999，这相当于对1000个值取平均，当$\beta$取0.9995，这就相当于对2000个值取平均。所以整个取值过程中，你需要更加密集地取值。

3. 超参数训练实战：Pandas vs. Caviar (Hyperparameters Tuning in Practice: Pandas vs. Caviar)

关于如何搜索超参数的问题，大概有两种重要的思想流派，或人们通常采用的两种重要但不同的方式。

3.1 照看一个模型 (Babysitting one model)

一种是你照看一个模型，通常是在有庞大的数据组，但没有许多计算资源或足够的CPU和GPU的前提下，基本而言，你只可以一次负担起试验一个模型或一小批模型，在这种情况下，即使当它在试验时，你也可以逐渐改良。比如，在第0天随机初始化参数，然后开始试验，然后你逐渐观察自己的学习曲线，可以是损失函数$J$ , data setting error or other things, gradually decrease in the first day, then at the end of the day, you may try to increase the learning rate a little bit, see how it will be, maybe the second day will find that it becomes better. Two days later, it's still doing pretty well, maybe fill inthe Momentumor reduce the variables. Then every day you watch it and keep tweaking your parameters. Maybe one day, you will find that your learning rate is too large, so you may fall back to the previous model, but you can say that you spend time looking after this model every day, even if it is over many days or many weeks. middle.
So it's an approach that people take care of a model, watch how it behaves, and patiently tune the learning rate, but that's usually the way people do it because you don't have enough computing power to experiment with a lot of models at the same time.

3.2 Simultaneous training of multiple models (Training many models in parallel)

Another way is to experiment with multiple models at the same time, you set some hyperparameters, let it run by itself, or for a day or even for several days, and then you will get a learning curve like the one above (blue curve), This could be loss function J, experimental error, or data error loss, but are all measures of the curve trajectory. Also you can start a different model with different hyperparameter settings, so your second model will generate a different learning curve (purple curve), maybe this one looks better. In the meantime, you can experiment with a third model, which may result in a new learning curve, or any other learning curve. Or you can test many different models in parallel at the same time, and the different orange curves correspond to different models. This way you can experiment with many different parameter settings and finally quickly choose the one that works best.

4. Normalizing Activations in a Network

4.1 The role of Batch normalization

After the rise of deep learning, one of the most important ideas is an algorithm called Batch normalization . Batch normalization will make your parameter search problem easier, make the choice of hyperparameters for the neural network more stable, the range of hyperparameters will be larger, work well, and will also make your training easier, Even the deep web.

When you train a neural network model, normalizing the input features can speed up the learning process. By calculating the mean value $m=\frac{1}{m}\sum x^{\left(i\right)}$ , then subtract the mean$x=x-\mu$ , calculate the variance$p^2=\frac{1}{m}\sum (x^{(i)}{)}^2$ , then normalize your input dataset$x=x/p^2$ . In the previous chapters, we learned that through normalization, the outline of the learning problem can be changed from flat to circular, which is easier for algorithm optimization. So for logistic regression and for normalizing the input eigenvalues ​​of neural networks, this is very efficient.

However, in the neural network training, we only normalized the input layer data, but did not normalize the middle layer . Although we have normalized the input data, but the input data after $s(W^{T}X+b)$ After such matrix multiplication and nonlinear operations, thedata distribution is likely to be changed, and with the multi-layer operation of the deep network, the data distribution will change more and more.
So if you want to train these parameters like$w^{[3]}，b^{\left[3\right]}$ , then normalize$a^{\left[2\right]}$ mean and variance, so that$w^{\left[3\right]}，b^{The\; training\; of\; [3]}$ is more efficient. Although strictly speaking, what we really normalize is not$a^{\left[2\right]}$ instead of$z^{[2]}$。深度学习文献中有一些争论，关于在激活函数之前是否应该将值$z^{[2]}$归一化，或是否应该在应用激活函数$a^{[2]}$后再规范值。实践中，经常做的是归一化$z^{[2]}$，我推荐其为默认选择。

4.2 Batch归一化的使用方法

在神经网络中，假设隐藏层单元$z^{[l](i)}$已知，其中$i\in [1,m]$。下面的公式都是针对第$l$层单元，但为了简化符号，因此省略$[l]$。
首先，取每个$z^{(i)}$值，使其规范化。方法如下，计算$z^{(i)}$的均值，再将每个$z^{(i)}$值减去均值，除以标准差。为了使数值稳定，通常在分母加上$ϵ$，以防$\sigma =0$的情况。 $$\mu =\frac{1}{m}\sum _{i=0}^m{z}^{(i)}$$ $${\sigma }^2=\frac{1}{m}\sum _{i=0}^m(z^{(i)}-\mu {)}^2$$ $$z_{norm}^{(i)}=\frac{z^{(i)}-}{\sqrt{p^2+ϵ}}$$The batch of data $z^{\left(i\right)}$ After normalization, it satisfies the normal distribution, at this timezzEach component of$z\; has\; a\; mean\; of\; 0\; and\; a\; variance\; of\; 1.$But the normalized$z$ will basically be limited to a normal distribution, which will reduce the expressiveness of the network, so all we need to do next isscale transformation and offset. $${\tilde{z}}^{\left(i\right)}=\gamma z_{norm}^{\left(i\right)}+\beta$$ Its intuitive function is to use$z_{norm}^{\left(i\right)}$乘以$\gamma$ adjusts the value size, plus$\beta$增加偏移后得到${\tilde{z}}^{(i)}$，这里的$\gamma$是尺度因子，$\beta$是平移因子。如果$\gamma =\sqrt{{\sigma }^2+ϵ}$（即$z_{norm}^{(i)}=\frac{z^{(i)}-\mu }{\sqrt{{\sigma }^2+ϵ}}$中的分母项），$\beta$等于$\mu$（这里的$\mu$就是$z_{norm}^{(i)}=\frac{z^{(i)}-\mu }{\sqrt{{\sigma }^2+ϵ}}$中的$\mu$），那么${\tilde{z}}^{(i)}=z^{(i)}$。因此，$\gamma z_{norm}^{(i)}+\beta \gamma$的实际作用在于，通过对$\gamma \text{γ}\gamma$和$\beta \text{β}\beta$ is reasonably set, and the hidden unit value z ( i ) \pmb{z^{(i)}} thatsatisfies other means and variances can be constructed$z^{(i)}\text{z(i)}{z}^{(i)}$。

5. Fitting Batch Normalization into a Neural Network (Fitting Batch Norm into a Neural Network)

In practice, Batch Normalization is often used with mini-batches of the training set . Assuming there is a neural network, the way you apply Batch normalization is to take the first mini-batch $X^{\left\{1\right\}}$ as input, then apply the parameter$w^{\left[1\right]}$ and$b^{\left[1\right]}$ Calculate$z^{\left[1\right]}$ , thenBatch normalizationwill subtract the mean, divided by the standard deviation, by$b^{\left[1\right]}$ and$c^{\left[1\right]}$ Rescale so that${\tilde{z}}^{\left[1\right]}$ , then apply the activation function$g^{[1]}({\tilde{z}}^{\left[1\right]})$to get$a^{\left[1\right]}$ . Then use the parameter$w^{\left[2\right]}$ and$b^{\left[2\right]}$ Calculate$z^{\left[2\right]}$ ,Batch normalizationto get${\tilde{z}}^{\left[2\right]}$ , introduce two parameters$b^{\left[2\right]}$ and$c^{\left[2\right]}$ , and finally apply the activation function$g^{[2]}({\tilde{z}}^{\left[2\right]})$to get$a^{\left[2\right]}$ . And so on, the second mini-batch$X^{\left\{2\right\}}$ , the third mini-batch$X^{\left\{3\right\}}$ and so on continue to train in this way. What needs to be emphasized
aboutBatch normalization

• Batch normalization occurs in the calculation of $zwaaa$ _$a$之间的。
• 这里的${\beta }^{[1]}，{\beta }^{[2]}$等和Momentum超参数$\beta$没有任何关系。
• $z$的计算方式是$z^{(i)}=w^T{a}^{(i)}+b$，而Batch归一化中均值的计算方式$\mu =\frac{1}{m}{\sum }_{i=0}^m{z}^{(i)}=E(w^T{a}^{(i)})+$b.znorm$(\; i\; )\; =\; z\; (\; i\; )\; -\; \mu \; \sigma \; 2\; +\; ϵ\; z\_\{norm\}^\{(i)\}=\backslash frac\{z^\{(i)\}\; -$$z_{norm}^{(i)}=\frac{z^{\left(i\right)}-}{\sqrt{p^2+ϵ}}$中$z^{(i)}-m=(w^T{a}^{\left(i\right)}+b)-\left(E\right(w^T{a}^{(i)})+b)=w^T{a}^{\left(i\right)}-E(w^T{a}^{\left(i\right)}$ . This means that regardless of the bias term$Whatever\; the\; value\; of\; b$ is, it will be eliminated. Therefore, the bias term b \pmb{b} can not be added when using Batch normalization$b\text{b}b$ , or set it to 0.

To summarize how to apply gradient descent using Batch Normalization . Assuming you are using mini-batch gradient descent , run $t\; is$ a for loop from 1 to the number of batches, and the following operations are performed in the for loop:

1. For mini-batch $X^{\left\{t\right\}}$ applies forward propagation, and applies forward propagation to each hidden layer, and calculates$z^{[l]}$；
2. Use Batch normalization to calculate ${\tilde{z}}^{\left[l\right]}$ , thus replacing$z^{\left[l\right]}$；
3. Use backpropagation to calculate the llthAll parameters of layer$l$$d{w}^{[l]}$、$d{b}^{\left[l\right]}$、$d{\beta }^{\left[l\right]}$和$d{\gamma }^{[l]}$ ；
4. Update parameters: $w^{[l]}=w^{\left[l\right]}-\alpha d{w}^{\left[l\right]}$，$b^{\left[l\right]}=b^{\left[l\right]}-ad{b}^{\left[l\right]}$，$c^{\left[l\right]}=c^{\left[l\right]}-\alpha d{\gamma }^{[l]}$。
   （Momentum、RMSprop、Adam等优化算法在这里也同样适用）

6. Batch Norm 为什么起作用? (Why does Batch Norm work?)

6.1 Covariate shift

假设有一个神经网络，建立在猫的识别检测上。假设你已经在所有黑猫的图像上训练了数据集，但是现在将此网络应用于有色猫的识别。在这种情况下，正样本中不只有黑猫，还有其它颜色的猫。

假设左图是黑猫的训练集正负样本分布情况，右图是黑猫和有色猫混合的训练集正负样本分布情况。在实际中，使用左图这样的训练集作为输入而训练结果不错的神经网络，同样给右图这样的训练集运行，却并不见得会好。即使存在运行都很好的同一个函数，也不希望去使用。
Covariate shift的概念就是由于训练集和测试集，即输入数据存在分布的差异性，给网络的泛化性和训练速度带来了影响。

6.2 Covariate shift如何影响神经网络?

现在有一个上图这样的深层神经网络，以隐藏层第三层为例，假设此网络已经学习了参数$w^{[3]}$和$b^{[3]}$。

Then mask the left part, from the perspective of the third hidden layer, it gets some values ​​from the previous layer, namely $a_1^{[2]}，a_2^{\left[2\right]}，a_3^{\left[2\right]}，a_4^{\left[2\right]}$, these values ​​can also be seen as input values $x_1，x_2，x_3，x_4$. The job of the third hidden layer is to find a way to map these values ​​to $\hat{y}$。

Now we uncover the left side of the network, this network also has parameters $w^{[1]}，b^{\left[1\right]}$ Sum$w^{[2]}，b^{\left[2\right]}$ , after each parameter update, after the network calculation of the first layer and the second layer of the hidden layer,$a_1^{[2]}，a_2^{\left[2\right]}，a_3^{\left[2\right]}，a_4^{\left[2\right]}$The value of will also change. For the third layer of the hidden layer, the distribution of the input data will change, so it has a Covariate shift problem.

6.3 How does Batch normalization solve the Covariate shift problem?

Draw the distribution of the hidden unit values ​​​​of the second layer (for ease of understanding, only $z_1^{\left[2\right]}$Sum $z_2^{\left[2\right]}$two hidden unit values), since $z_1^{\left[2\right]}$Sum $z_2^{\left[2\right]}$The value of will change, and therefore its data distribution will also change.
What Batch Normalization does is it limits how much the distribution of these hidden unit values ​​varies. When the neural network updates the parameters in the previous layer, Batch normalization can ensure that no matter how it changes $z_1^{\left[2\right]}$Sum $z_2^{\left[2\right]}$The mean and variance of remain unchanged (the mean and variance can be 0 and 1 respectively, that is, conform to the normal distribution, or can be determined by the parameter $\beta$和$\gamma$决定）。
Batch归一化减少了输入值改变的问题，使输入值变得更稳定，神经网络中后面的层就会有更坚实的基础。即使输入分布改变了一些，也会改变得更少。Batch归一化让当前层保持学习，当输入分布改变时，迫使后面的层对其适应的程度减小了，或者说它减弱了前层参数的作用与后层参数的作用之间的联系，使得网络中每一层都可以自己学习，稍稍独立于其它层，这有助于加速整个网络的学习。

6.4 Batch归一化的其他作用

Batch归一化还有一个作用，它有轻微的正则化效果。Batch归一化通常与mini-batch梯度下降法一起使用，由于每个mini-batch X { t } X^{\{t\}} X{ t}相当于不同的输入数据集，且mini-batch size较小，因此在mini-batch上计算出的均值和方差会有一些小的噪声。同时从$z^{[l]}$到${\tilde{z}}^{[l]}$的缩放过程中也有一些噪音，因为它是用本身存在噪音的均值和方差计算得出的。
所以和dropout相似，Batch归一化给每个隐藏层的激活值上增加了噪音，这迫使后面的单元不过分依赖任何一个隐藏单元。对于隐藏单元来说，向输入添加方差极小的噪声等价于对权重施加范数惩罚，因此相当于有正则化的作用。但是因为添加的噪音很微小，所以正则化的效果并不是很大。
如果你想得到dropout更强大的正则化效果，你可以将Batch归一化和dropout一起使用。另外，通过应用较大的min-batch，可以减少噪音，同时也减少正则化效果。但事实上，不建议把Batch归一化当作正则化。

7. 测试时的 Batch Norm (Batch Norm at Test Time)

Batch归一化将你的数据以mini-batch的形式逐一处理，但在测试时，你可能需要对每个样本逐一处理。

First review the process of performing Batch normalization in the form of mini-batch during the training phase . In a mini-batch , first all samples z ( i ) z^{(i)} in the mini-batch$z^{\left(i\right)}$ The value is accumulated and summed to calculate the mean value, here$m$ to represent the number of samples in thismini-batch, not the entire training set. Then calculate the variance, and then calculate$z_{norm}^{(i)}$, that is, adjusted with the mean and standard deviation, the denominator plus $ϵ$ is for numerical stability. Finally, use$c$和$\beta$ adjusts znormagain$z_{norm}$get $\tilde{z}$。
请注意在训练阶段用于调节计算的均值$\mu$和方差${\sigma }^2$是在整个mini-batch上进行计算，但是在测试阶段，你也许不能对一个mini-batch中所有的样本同时处理，因此你需要用其它方式来得到均值$\mu$和方差${\sigma }^2$，而且如果你只有一个样本，一个样本的均值和方差没有意义。在典型的Batch归一化应用中，你需要用一个指数加权平均来估算，这个平均数涵盖了所有mini-batch。
假设在第$l$层，首先训练第一个mini-batch X { 1 } X^{\{1\}} X{ 1}，得到第一个mini-batch的均值 μ { 1 } [ l ] \mu^{\{1\}[l]} μ{ 1}[l]。然后在这一层继续训练第二个mini-batch X { 2 } X^{\{2\}} X{ 2}，得到均值 μ { 2 } [ l ] \mu^{\{2\}[l]} μ{ 2}[l]。接着在这一层训练第三个mini-batch X { 3 } X^{\{3\}} X{ 3}，得到均值 μ { 3 } [ l ] \mu^{\{3\}[l]} μ{ 3}[l]。正如之前用的指数加权平均来计算${\theta }_1，{\theta }_2，{\theta }_3$The mean value of , here you can also use the exponential weighted average to estimate the hidden units $mean\; of\; z$ . Similarly, you can also use the exponentially weighted average to trackσ 2 \sigma^2inmini-batch$p^2$ . Therefore, while training the neural network with differentmini-batchesthe μ \muof each layer you are looking at$\mu$和$p^{The\; real-time\; value\; of\; the\; mean\; of\; 2}$ .
Finally, when testing, the corresponding equation$z_{norm}^{(i)}=\frac{z^{\left(i\right)}-}{\sqrt{p^2+ϵ}}$, you just need to use your $z$ value to calculate$z_{norm}^{\left(i\right)}$，用$\mu$和$p^2$ exponentially weighted average to make adjustments, and then use the just calculated$z_{norm}$and the parameter β \beta you got during neural network training$\beta$和$\gamma$ to compute z ~ \tilde{z}for that test sample$\tilde{z}$ value.

8. Softmax Regression

All the examples of classification we have encountered so far have used binary classification, which has only two possible labels, 0 or 1. There is a general form of logistic regression called Softmax regression , which is used to solve multi-classification problems.

Suppose you need to identify cats, dogs and chickens. Here I set cats as category 1, dogs as category 2, and chickens as category 3. If they do not belong to any of the above categories, set them as category 0 (as shown above). The symbol CC is used here$C$ represents the total number of categories in the classification.

First build a neural network with 4 output layers, or $C$ output units, so$n^{\left[L\right]}$ , that is, the number of output layer units is equal to 4, or in general equal to$C$。

Each output unit of the output layer will give the probability of each category (as shown in the figure above), that is, when the input is $When\; X$ , the first node corresponds to the probability$P(other|X)$ , the second node corresponds to the probability that the output is a cat (category 1)$P(cat|X)$ , the third node corresponds to the probability that the output is a dog (category 2)$P(dog|X)$ , the fourth node corresponds to the probability that the output is a chick (category 3)$P(babychickk|X)$ . Therefore$\hat{y}$will be a 4×1 dimensional vector, and the probabilities of the four outputs should add up to 1.
Making your network do the above requires a Softmax layer, and an output layer to generate the output. In the last layer of the neural network, you will calculate the linear part of each layer as usual, the calculation process is as follows:
(1) Calculate $z^{[L]}=W^{[L]}{a}^{[L-1]}+b^{\left[L\right]}$
(2) ApplySoftmaxactivation function:

1. Calculate a temporary variable $t=e^{z^{[L]}}$，这是对所有元素求幂。其中$z^{[L]}$的维度是4×1，因此$t=e^{z^{[l]}}$是一个4×1维向量；
2. 对$t$进行归一化，使输出的和为1，然后输出$a^{[L]}$。因此$a^{[L]}=\frac{e^{z^{[L]}}}{{\sum }_{j=1}^4{t}_i}$，换句话说，$a^{[L]}$也是一个4×1维向量，而这个四维向量的第$i$个元素$a_i^{[L]}=\frac{t_i}{{\sum }_{j=1}^4{t}_i}$。

To give a specific example, suppose $z^{[L]}=\left[\begin{array}{c}5\\ 2\\ -1\\ 3\end{array}\right]$, and then use the element exponentiation method to calculate $t=e^{z^{\left[L\right]}}=\left[\begin{array}{c}{e}^5\\ e^2\\ e^{-1}\\ e^3\end{array}\right]=\left[\begin{array}{c}148.4\\ 7.4\\ 0.4\\ 20.1\end{array}\right]$。从向量$t$到向量$a^{[l]}$只需要将这些元素归一化，使总和为1，即把向量$t$中的所有元素加起来，得到176.3，最终$a^{[l]}=\hat{y}=\frac{t}{176.3}=\left[\begin{array}{c}0.841\\ 0.041\\ 0.002\\ 0.114\end{array}\right]$。因此在输出层，第一个节点的输出是0.842，即输入$X$为类别0的概率是84.2%。同理，第二个节点的输出是0.041，即输入$X$为类别1的概率是4.1%。第三个节点的输出是0.002，即输入$X$为类别2的概率是0.2%。第四个节点的输出是0.114，即输入$X$为类别3的概率是11.4%。

9. 训练一个 Softmax 分类器 (Training a Softmax Classifier)

关于训练带有Softmax输出层的神经网络，具体而言，我们先定义训练神经网络会用到的损失函数。以上一章节为例，假设输入是一张猫（类别1）的图片，即真实标签是$y=\left[\begin{array}{c}0\\ 1\\ 0\\ 0\end{array}\right]$。假设你的神经网络输出的是$\hat{y}=a^{[L]}=\left[\begin{array}{c}0.3\\ 0.2\\ 0.1\\ 0.4\end{array}\right]$, where $\hat{y}$is a vector whose elements sum to 1. The neural network does not perform well for this sample, which is actually a cat, but only assigns a 20% probability of being a cat, so it does not perform well in this case.
In Softmax classification, the loss function we generally use is $$L(\hat{y},y)=-\sum _{j=1}^4{y}_j\log {\hat{y}}_j$$In this sample $y_1=y_3=y_4=0$，$y_2=1$，因此损失函数$L(\hat{y},y)=-y_{2}t\log {\hat{y}}_2=-\log {\hat{y}}_2$。$L(\hat{y},y)=-{\sum }_{j=1}^4{y}_j\log {\hat{y}}_j=-y_2\log {\hat{y}}_2=-\log {\hat{y}}_2$. This means that if your learning algorithm tries to make it smaller, because the gradient descent method is used to reduce the loss of the training set, let the loss function L ( y ^ , y ) L(\hat{y} $L(\hat{y},y)$ becomes smaller, that is, let$-\log {\hat{y}}_2-$ becomes smaller, and${\hat{y}}_2$Make it as big as possible.
In a nutshell, what the loss function does is find the true class in your training set, and then try to make the corresponding probability of that class as high as possible.
This is the loss for a single training sample, the loss for the entire training set $J$ is to add up the predictions of all training samples, that is,$$J(w^{[1]},b^{\left[1\right]},\cdots )=\frac{1}{m}\sum _{i=1}^{m}L({\hat{y}}^{(i)},y^{\left(i\right)})$$So what you have to do is use the gradient descent method to minimize the loss here. In a neural network withSoftmaxoutput layer, the process of implementing the gradient descent method is as follows:

• Forward propagation process:
  • ① The output layer will calculate $z^{\left[l\right]}$ , whose dimension is C×1.
  • ② Use the Softmax activation function to get $a^{\left[l\right]}$ , or$\hat{y}$。
  • ③ Calculate the loss.
• Backpropagation process:
  • ① $d{z}^{[l]}=\hat{y}-y$ , which is for$z^{\left[l\right]}$ The partial derivative of the loss function$d{z}^{\left[l\right]}=\frac{\partial J}{\partial z^{\left[l\right]}}$。
  • ② Calculate all derivatives required in the entire neural network.