Backpropagating errors from multiple nodes
The figure below shows a neural network consisting of two input nodes and two output nodes.
Each output node has an error. In fact, when we do not train the neural network, the output node must have errors. You can see that each error is fed back to the weights in the neural network for updates to the weight values. We can use the method in the previous subsection, where we divide the error in proportion to the weight of its contributing nodes.
In fact, when the neural network has more than two output nodes, the calculation method of its weight update is still the same. We only need to verify the calculation method of the first output node, and update the error of subsequent nodes to the weight of its contributing nodes. Why is it so simple? Because the contribution nodes of each output node in the neural network are independent of each other, that is to say, the contribution node of a certain node does not depend on the contribution nodes of other output nodes.
Looking carefully at the above figure, we put the error flag of the first output node at e1. Remember that this is the difference between the actual output o1 and the training value t1 we expect to output, i.e. e1=t1-o1. The error of the second output node we label e2.
From the figure, we can observe that the error e1 is divided according to the weight ratio of its connected contributing nodes, and the weight values of the contributing nodes are w11 and w21 respectively. Similarly, the error e2 will be divided according to the weights w12 and w22.
Let's write the error split value size so we don't get misunderstandings. The error e1 is used to update the weights. So the error scale used to update the weight w11 should be:
Similarly, the ratio of e1 to update weight w21 is:
At first glance, this ratio is a little confusing. Let's actually calculate its size.
If w11 is twice larger than w21, for example, w11=6, w21=3 Therefore, the ratio used to update w11 is 6/6+3=2/3, and the ratio used to update w21 is 1/3. We can actually calculate what proportion is used to update w21? 3/6+3=1/3, it is indeed 1/3 to update.
If the weights of the contributing nodes are equal, then the proportion of updating the corresponding weights with the error should also be equal. Suppose, w11=w21=4, then the weights of the weight contribution nodes will be updated in the proportion of 4/4+4=1/2.
Before moving on to the next calculation, let's stop and think about what we've actually done. We know that we need to use the error to update certain parameters in the neural network, such as the weights between connected nodes. We already know how to use the output of the neural network to update the weights between nodes. We also know how to use the output node to update the weight value between the contributing nodes when there is more than one output node in the neural network: we just calculate how each output node updates the weight between its contributing nodes in the same way. value. very good!
What happens when there are more than two layers in a neural network? How do we update the weights between the contributing nodes from the output node?