What is bp neural network, neural network and bp neural network

1. What is BP neural network?

The basic idea of the BP algorithm is: the learning process is composed of two parts: signal forward propagation and error backpropagation; during forward propagation, the input samples are passed in from the input layer, processed layer by layer through each hidden layer, and transmitted to the In the output layer, if the output of the output layer does not match the expectation, the error is sent back layer by layer as an adjustment signal, and the connection weight matrix between neurons is processed to reduce the error. After repeated learning, the error is finally reduced to an acceptable range. The specific steps are as follows:
1. Take a certain sample from the training set and input the information into the network.
2. The actual output of the neural network is obtained after forward layer-by-layer processing of the connection between each node.
3. Calculate the error between the actual output of the network and the expected output.
4. The error is reversely transmitted layer by layer to the previous layers, and the error signal is loaded on the connection weight according to a certain principle, so that the connection weight of the entire neural network is converted to the direction of error reduction.
5. Repeat the above steps for each input-output sample pair in the training set until the error of the entire training sample set is reduced to meet the requirements.

Google AI Writing Project: Little Fat Cat

2. bp neural network

The BP (Back Propagation) network was proposed by a team of scientists headed by Rumelhart and McCelland in 1986. It is a multi-layer feed-forward network trained by the error back propagation algorithm. It is one of the most widely used neural network models at present. The bp neural network is Is it a kind of dnn network ? The BP network can learn and store a large number of input-output pattern mapping relationships without revealing the mathematical equations describing the mapping relationship in advance. Its learning rule is to use the steepest descent method to continuously adjust the weights and thresholds of the network through backpropagation to minimize the sum of squared errors of the network. The topological structure of BP neural network model includes input layer (input), hidden layer (hide layer) and output layer (output layer).
Artificial neural network is the second way of simulating human thinking. This is a nonlinear dynamical system characterized by distributed storage and parallel collaborative processing of information. Although the structure of a single neuron is extremely simple and its functions are limited, the behaviors that can be realized by a network system composed of a large number of neurons are extremely colorful.
Artificial neural networks must first learn with certain learning criteria before they can work. Now take the recognition of handwritten "A" and "B" by artificial neural network as an example. It is stipulated that when "A" is input into the network, "1" should be output, and when the input is "B", the output is "0".
Therefore, the criterion of network learning should be: if the network makes a wrong judgment, the network learning should reduce the possibility of the network making the same mistake next time. First, assign random values in the interval (0, 1) to each connection weight of the network, input the image mode corresponding to "A" to the network, and the network will weight the sum of the input modes, compare them with the threshold, and then perform a Linear operation to get the output of the network. In this case, the probability of the network output being "1" and "0" is 50% each, which means it is completely random. At this time, if the output is "1" (the result is correct), the connection weight is increased so that the network can still make a correct judgment when it encounters the "A" mode input again.
If the output is "0" (that is, the result is wrong), adjust the weight of the network connection towards the direction of reducing the weight of the comprehensive input. Same possibility of error. With such operation and adjustment, when several handwritten letters "A" and "B" are input to the network in turn, after several times of learning through the network according to the above learning method, the correct rate of network judgment will be greatly improved. This shows that the learning of these two modes by the network has been successful, and it has memorized these two modes on the connection weights of the network in a distributed manner. When the network encounters any of these patterns again, it can make a quick and accurate judgment and identification. Generally speaking, the more neurons in the network, the more patterns it can remember and recognize.
The single-hidden layer feedforward network with the topology shown in the figure is generally called a three-layer feedforward network or a three-layer perceptron, namely: input layer, middle layer (also called hidden layer) and output layer. Its characteristics are: the neurons in each layer are only fully connected with the neurons in the adjacent layer, there is no connection between neurons in the same layer, and there is no feedback connection between neurons in each layer, forming a feedforward type with a hierarchical structure. neural network system. A single-layer feedforward neural network can only solve linearly separable problems, and a network that can solve nonlinear problems must be a multi-layer neural network with hidden layers.
The research content of neural network is quite extensive, reflecting the characteristics of multidisciplinary interdisciplinary technical fields. The main research work focuses on the following aspects:
(1) Biological prototype research. Study the biological prototype structure and functional mechanism of nerve cells, neural networks, and nervous systems from the aspects of physiology, psychology, anatomy, brain science, pathology and other biological sciences.
(2) Establish a theoretical model. Based on the study of biological prototypes, the theoretical models of neurons and neural networks are established. These include conceptual models, knowledge models, physical and chemical models, mathematical models, etc.
(3) Network model and algorithm research. Construct a specific neural network model on the basis of theoretical model research to realize computer simulation or prepare hardware, including research on network learning algorithms. Work in this area is also known as technical model research.
(4) Artificial neural network application system. On the basis of network model and algorithm research, artificial neural networks are used to form practical application systems, for example, to complete certain signal processing or pattern recognition functions, to construct expert systems, to make robots, and so on.
Throughout the development history of contemporary emerging science and technology, human beings have gone through bumpy roads in the process of conquering space, elementary particles, origin of life and other scientific and technological fields. We will also see that research exploring the function of the human brain and neural networks will advance with each passing day as difficulties are overcome.
Neural networks can be used for classification, clustering, prediction, etc. The neural network needs a certain amount of historical data. Through the training of historical data, the network can learn the hidden knowledge in the data. In your problem, you must first find some characteristics of certain problems and the corresponding evaluation data, and use these data to train the neural network.
Although the BP network has been widely used, it also has some defects and deficiencies, mainly including the following problems.
First, since the learning rate is fixed, the network converges slowly and requires a long training time. For some complex problems, the training time required by the BP algorithm may be very long, which is mainly caused by the too small learning rate, which can be improved by using a variable learning rate or an adaptive learning rate.
Secondly, the BP algorithm can make the weight converge to a certain value, but it does not guarantee that it is the global minimum of the error plane, because the gradient descent method may produce a local minimum. For this problem, the additional momentum method can be used to solve it.
Thirdly, there is no theoretical guidance for the selection of the number of layers and the number of units in the hidden layer of the network, and it is generally determined based on experience or through repeated experiments. Therefore, there is often great redundancy in the network, which also increases the burden of network learning to a certain extent.
Finally, the learning and memory of the network is unstable. In other words, if learning samples are added, the trained network needs to be trained from scratch, and there is no memory for the previous weights and thresholds. However, better weights for prediction, classification or clustering can be saved.

3. BP artificial neural network

Artificial neural network (ANN) refers to a network formed by a large number of neurons similar to the natural nervous system. It is a type of artificial system that uses engineering technology to simulate the structural and functional characteristics of biological networks. The neural network not only has the general computing ability to process numerical data, but also has the thinking, learning, and memory abilities to process knowledge. It uses a method similar to "black box" to find out the difference between input and output variables through learning and memory. Linear relationship (mapping), when executing problems and solutions, input the acquired data into the trained network, perform network reasoning based on the knowledge learned by the network, and obtain reasonable answers and results.

Many problems in geotechnical engineering are nonlinear problems, and the relationship between variables is very complicated, which is difficult to describe with exact mathematical and mechanical models. The representativeness of the measured data on the engineering site is related to the location, scope and means of the measuring point. Sometimes it is difficult to meet the statistical conditions and laws required by traditional statistical methods. In addition, the complexity and uncertainty of geotechnical engineering information, so the use of neural network The method is suitable for solving geotechnical problems.

The BP neural network model is the abbreviation of the error backpropagation (BackPagation) network model. It consists of input layer, hidden layer and output layer. The learning process of the network is the process of gradually modifying the connection weights between nodes in each layer of the network. This process consists of two parts: forward propagation and back propagation. Forward propagation is the transmission of the input pattern from the input layer to the output layer through hidden layer processing; back propagation is the propagation of the mean square error information from the output layer to the input layer, returning the error signal along the original connection path, and modifying the neural network of each layer. The weight of the element minimizes the error signal.

In the process of establishing and applying the BP neural network model, there are mainly deficiencies and suggestions in the following four aspects:

(1) For the neural network, the more data there is, the better the training effect of the network is, and it can better reflect the reality. However, in actual operation, due to the limitation of conditions, it is difficult to select a large number of sample values for training, and the number of samples is relatively small.

(2) The calculation speed of the BP network model is slow, and it cannot express the relationship between the predicted quantity and its related parameters.

(3) Establish a model based on quantitative data. If sufficient data can be collected, use qualitative indicators (such as foundation pit precipitation mode, foundation pit support mode, construction conditions, etc.) and some easily obtained quantitative indicators as the input layer. With the evaluation grade as the output layer, the BP network model established in this way will be more accurate and comprehensive.

(4) The BP artificial neural network system has the characteristics of nonlinearity and intelligence. The qualitative description and quantitative calculation, precise logical analysis and non-deterministic reasoning are well considered, but due to different samples, the weights of influencing factors are different, and the qualitative parameters are quantified based on prior knowledge and previous experience. It will inevitably affect the objectivity and accuracy of the evaluation. Therefore, in the actual evaluation, only by selecting different analysis indicators according to different foundation pit construction conditions, different surrounding environmental conditions, and the needs of different users, can we meet the requirements of geological environment evaluation under complex working conditions and achieve better results. application effect.

4. What is the BP network?

Our most commonly used neural network is the BP network, also known as a multi-layer feedforward network. BP is written by back propagation, which means back propagation. I was quite confused before, because I never understood why it was called a feedforward network for a while, and a BP (back propagation) network for a while. Isn’t it a contradiction? In fact, this is the case. Feedforward is based on the network structure. The neurons in the previous layer feed into the neurons in the next layer in one direction, and the neurons in the back do not feed back to the previous neurons; while the BP network is from the network. In terms of the training method, it means that the training algorithm of the network is the backpropagation algorithm, that is, the training of the link weights of the neurons starts from the last layer (output layer), and then reversely updates the link weights of the previous layer in turn. . So the two are not contradictory, it's just that I don't understand the essence of it.
Casually mentioning the powerful power of BP network:
1) Any Boolean function can be accurately represented by a network of two-layer units, but the number of hidden layer neurons required grows exponentially with the number of network inputs; 2) Any continuous
function can be approximated with arbitrary precision by a two-layer network. The two-layer network here means that the hidden layer uses a sigmoid unit and the output layer uses a non-threshold linear unit;
3) Any function can be approximated by a three-layer network with arbitrary precision. Its two hidden layers use sigmoid units, and the output layer uses non-thresholded linear units.
Reference materials: [Note] Reference from "Machine Learning"

5. Principle of BP neural network

There are many models of artificial neural network, but the most widely used, the most intuitive basic idea, and the easiest to understand is the multi-layer feedforward neural network and error back-propagation learning algorithm (Error Back-Prooaeation), referred to as BP network.

In the book "Parallel Distributed Processing" published by scientists led by Rumelhart and McCelland in 1986, the error backpropagation learning algorithm was completely proposed and widely accepted. A multi-layer perceptual network is a hierarchical neural network with three or more layers. A typical multi-layer perceptual network is a three-layer, feed-forward hierarchical network (Figure 4.1), namely: input layer, hidden layer (also called intermediate layer), and output layer, as follows:

Figure 4.1 Three-layer BP network structure

(1) Input layer

The input layer is the interface for the network to interact with the outside world. Generally, the input layer is only the storage layer of the input vector, and it does not perform any processing and processing on the input vector. The number of neurons in the input layer can be determined according to the problem to be solved and the way of data representation. Generally speaking, if the input vector is an image, the number of neurons in the input layer can be the number of pixels of the image, or the number of processed image features.

(2) hidden layer

In 1989, Robert Hecht Nielsno proved that any continuous function in a closed interval can be approximated by a hidden layer BP network, so a three-layer BP network can complete any n-dimensional to m-dimensional mapping. Although increasing the number of hidden layers can further reduce the error and improve the accuracy, it also complicates the network, thus increasing the training time of the network weights. The improvement of error accuracy can also be achieved by increasing the number of neurons in the hidden layer, and the training effect is easier to observe and adjust than increasing the number of hidden layers, so in general, priority should be given to increasing the number of neurons in the hidden layer , and then choose the appropriate number of hidden layers according to the specific situation.

(3) Output layer

The output layer outputs the result vector of network training. The dimension of the output vector should be designed according to the specific application requirements. When designing, the scale of the system should be reduced as much as possible to reduce the complexity of the system. If the network is used as a recognizer, the recognized class neuron is close to 1, while other neuron outputs are close to 0.

The neurons between adjacent layers of the above three-layer network are fully connected, that is, every neuron in the next layer is fully connected with every neuron in the previous layer, and there is no connection between neurons in each layer. The connection and connection strength constitute the weight matrix W of the network.

The BP network is learned in a way taught by a teacher. First, the teacher sets an expected output value for each input mode. Then input the actual learning and memory pattern to the network, and propagate from the input layer to the output layer through the middle layer (called "pattern forward propagation"). The difference between the actual output and the expected output is the error. According to the rule of minimum square error, the connection weights are corrected layer by layer from the output layer to the middle layer. This process is called "error backpropagation" (Chen Zhengchang, 2005). Therefore, the error backpropagation neural network is also referred to as BP (Back Propagation) network. The process of "mode forward propagation" and "error reverse propagation" is repeated alternately. The actual output of the network is gradually approaching the corresponding expected output, and the correct rate of the network's response to the input mode is also increasing. Through this learning process, after determining the connection weights between the next layers. The typical three-layer BP neural network learning and program running process are as follows (Zhao Yuan, 2006):

(1) First, explain the form and meaning of each symbol:

Network input vector Pk = (a1, a2, ..., an);

Network target vector Tk = (y1, y2, ..., yn);

The input vector Sk=(s1, s2,..., sp) of the middle layer unit, the output vector Bk=(b1, b2,..., bp);

Output layer unit input vector Lk = (l1, l2, ..., lq), output vector Ck = (c1, c2, ..., cq);

The connection weight wij from the input layer to the middle layer, i=1, 2,..., n, j=1, 2,...p;

The connection weight vjt from the middle layer to the output layer, j=1, 2,..., p, t=1, 2,..., p;

The output threshold θj of each unit in the middle layer, j=1, 2,..., p;

The output threshold γj of each unit in the output layer, j=1, 2, ..., p;

Parameter k=1, 2, . . . , m.

(2) Initialization. Assign random values within the interval (-1, 1) to each connection weight wij, vjt, threshold θj and γj.

(3) Randomly select a set of input and target samples

provided to the network.

(4) Use the input sample

, connection weight wij and threshold θj to calculate the input sj of each unit in the middle layer, and then use sj to calculate the output bj of each unit in the middle layer through the transfer function.