Zhang Yuhong amount extracted from the "depth study of the United States", the book is really good, oh!
I felt like distributed before such words, listening unfathomable, so to see, so I will not continue reading interest in the title containing words distributed, but Zhang Yuhong of this book is really great, deep was my heart, every point of his right knowledge, people indulge!
Before multilayer feedforward neural network, the input layer neurons mainly for inputting additional information received in the hidden layer and output layer has a built-in function is activated, processing may be performed on the input signal, the output from the final result layer presented.
It is noted that, in the activation function of neurons is not limited to the step function (SGN) we mentioned earlier, the Sigmoid function, may also be used in the present study depth ReLU (Rectified Linear Unit, linearity correction means) cROSS loss function and angry "So bit max return" and the like.
In simple terms, the process of learning neural network, is to adjust the width of the output value of the connection weights between neurons, and each neuron function through the training data. In other words, the neural network to learn something, it is implicit in the connection weights and width values.
Anthropomorphic, for the identification of an object, the neural network connection weights and width values, it is about the subject of "Memories"! We know that the brain's memory for things and concepts, not stored in a single place, but distributed to exist in a vast network of neurons.
Silicon Valley investors believe Mr. Wang Chuan, distributed characterization (Distributed Representation) is a core idea of artificial neural network research. So what is distributed to characterize it? Simply put, when we express a concept, and the concept is not between neurons-one correspondence map (map) is stored, the relationship between them is many to many. Specifically, a concept that can be defined by a plurality of neurons common expression, while a neuron may also be involved in the expression of a number of different concepts, but the share of the weight difference in terminology.
For example, for the concept of "little red car", if the distributed features to express, there may be a neuron represents the size (shape: small), a neuron represents the color (color: red), as well as a nerve yuan on behalf of the car category (category: cars). Only when these three neurons are activated simultaneously, it can more accurately describe the object we want to express.
Distributed characterization represents has many advantages. The most important point than when a partial malfunction of neurons, the expression of information, there is destruction of the damage. For example, we often see in film and television work in such a scenario, enemies meet exceptionally jealous, a person (A) angry and said, "You ashes, I know you (B)!" This is not true to say B "ashes", but rather said that although the passage of time, people have changed, the appearance of party B has also changed a lot (to identify people who a, B information stored in their brain is incomplete), but it does not matter as long as part of the core features of B still, that a or B to be able to recognize clearly, so true!
In fact, the use of a distributed neural network characteristic expression, can also be used to prevent the occurrence of over-fitting. In 2012, Professor Hinton published a highly cited paper [Improving neural networks by preventing co-adaptation of feature detectors [J] Computer Science, 2012. ] , Which refers to an in depth study of the widely used techniques: discard learning (Dropout learning). The core idea of the foregoing and explain the concept of the algorithm has the same purpose.