"Principles of Artificial Neural Networks" Reading Notes (8) Design and Development of Artificial Neural Network Applications

Summary of all notes: "Principles of Artificial Neural Networks"-Summary of Reading Notes

1. The characteristics and scope of application of artificial neural network

Characteristics of artificial neural network applications

• The application designs a batch of dense data, and these data depend on multiple interrelated parameters;
• In the field of the application design, and accumulated or collected a large amount of historical data and samples;
• The data provided in this application is imperfect and error-prone, but they describe a certain sample pattern;
• The problems involved in this application either have no definite solution method, or it takes too much time and too long to obtain a certain solution method.

The principle of choosing ANN technology

• Artificial neural network technology has certain advantages in dealing with more complicated problems that do not know much about internal rules, and cannot be described by mathematical expressions such as a set of rules and equations;
• The mathematical algorithm of the problem lacks clear analysis and analysis, which cannot be solved by ordinary computer methods or is very difficult to solve;
• The problem requires qualitative description or complex quantitative inference;
• The solution of the problem is derived from some highly correlated parameters, and the parameters do not have exact quantitative representations;
• Although the data is available, it is changeable, fuzzy, noisy or error, and uncertain;
• Have enough data accumulation to produce a sufficient set of training and test patterns;
• The development time required by the project is too short, but the time is sufficient for training the neural network.

2. The design and development process of artificial neural network

3. Selection of Artificial Neural Network Model

The size of the artificial neural network

• Non-statistical network models, such as BP neural network, should generally have less than 1000 nodes;
• Statistical network models, such as Boltzmann machines, generally have less than 200 nodes.

Required output type

• classification
• Graphics
• Real number
• optimize

Types of associative memory

• Self-association
• Connected Thinking

Training method

• Guided learning and training
• Unsupervised learning and training

Time limit

• Training time of neural network
• execution time

Fourth, the design of artificial neural network model

Node-level design: determine the neuron node type and transfer function of the selected artificial neural network model;

Network-level design: Determine the size, number of layers, number of nodes, and interconnection relationships between nodes of the neural network;

Training level design: Determine the learning and training algorithm, learning parameters, initial conditions and initial values ​​used by the neural network.

Node-level design

The combination of neuron node input signals

• Weighted sum

Transfer function used by neuron node

Network-level design

The number of layers of the neural network

• Most common artificial neural network models have a predetermined number of layers
• Hidden layer of BP neural network
  Experience shows that for classification or discrimination boundary problems (including binary input and output logic and decision boundary problems), generally only one hidden layer is needed.
  But if the output of the neural network needs to be expressed as an arbitrary continuous function of the input or some complex functions need to be expressed, two hidden layers can be used.

Number of input layer nodes

• The number of nodes in the input layer depends on the dimensions of the source data, and each dimension corresponds to an input node;
• Preprocessing of source data;
• The representation method of the source data.

Number of nodes in the hidden layer

• Increasing the number of nodes in the hidden layer can improve the accuracy of matching the neural network with the training data (the accuracy is approximately proportional to the square of the number of nodes in the hidden layer);
• In order to improve the generalization reasoning ability of the neural network, that is, to improve the adaptability of the neural network to new graphs, it is required to appropriately reduce the number of nodes in the hidden layer.

Empirical formula

• Nielsen proposed but the number of hidden layer nodes is $p=2n+1$ , where$p$ is the number of hidden layer nodes,$n$ is the number of nodes in the input layer;
• Lipmann believes that the maximum number of hidden layer nodes is $p=(n+1)\ast q$，$q$ is the number of nodes in the output layer;
• Kuarycki believes that the maximum number of hidden layer nodes is $p=3\ast q$；
• Maren et al. believe that for small networks, when the number of input nodes is greater than the number of output nodes, the maximum number of hidden layer nodes is $p=\sqrt{n\ast q}$。

Lippmann believes that in a neural network model with two hidden layers used for image recognition, the number of nodes in the second hidden layer is $p_2=2\ast q$ , where$p_2$Is the number of nodes in the second hidden layer, $q$ is the number of nodes in the output layer.

According to experiments, Kuarychi found that in high-dimensional input, the ratio of the optimal number of nodes from the first hidden layer to the second hidden layer is 3:1.

Gorman and Sejnowski proposed the formula for estimating the number of nodes in the hidden layer of a neural network for pattern classification: $p={\log }_{2}m$ , where$m$ is the number of input training patterns.

Nielsenhe Illingworth suggested that the formula for estimating the number of hidden layer nodes of the neural network used for process control is $p=4\ast q$。

Number of output layer nodes

• The number of nodes in the output layer is determined by the adopted artificial neural network model and the required output representation;
• The number of output nodes of the artificial neural network model with self-association ability is the same as the number of input nodes;
• The number of output nodes of artificial neural network models with interconnection ability is generally less than the number of input nodes.

Connection between nodes

• The connection relationship between nodes reflects the interconnection structure of the artificial neural network model.

Training level design

Learning training algorithm

• The existing ANN models basically have certain learning and training algorithms, but in practical applications, the adopted learning and training algorithms can be appropriately adjusted and improved according to specific needs.

Learning parameters and initial conditions

• Without certain theoretical guidance, the commonly used methods are also based on empirical data and constantly tested.

Termination condition

• It depends on the accuracy required by the application, the number of learning times or the learning time.

Five, the realization of artificial neural network model

Prepare sample data

Collection and selection of sample data

Type and representation of sample data

• Numerical data
• Language data: converted to numerical data by preprocessing

Preprocessing of sample data

• 归一化处理$$\begin{gathered} {\hat{x}}_i=\frac{x_i-x_{min}}{x_{max}-x_{min}} \\ {x}_{mid}=\frac{x_{max}+x_{min}}{2} \\ {\hat{x}}_i=\frac{x_i-x_{mid}}{\frac{1}{2}(x_{max}-x_{min})} \end{gathered}$$
• Distributed processing

Select training samples

Determination of the number of training samples

• Rule of thumb: The number of training samples is 5 ∼ 10 5\sim 10 of the total number of network connection weights$5\sim 10$ times
  

Selection and organization of training samples

• Equalize sample categories, cross input samples of different categories, or randomly select input samples from the training set.

Selection of training samples and test samples

• Among a large number of relevant data samples collected, 70% ∼ 80% 70\%\sim80\% can be randomly selected$70\%\sim .\; 80\%$ of the samples as the training data, the remaining$20\%\sim .\; 30\%$ data as the test sample.

Network training and testing

The size and speed of neural networks

Full training of the neural network

Neural network performance

• The correct judgment ratio of the neural network to the experimental data should reach $95\%$ or more;
• The best number of training sessions.

The next chapter Portal: "Principles of Artificial Neural Networks" Reading Notes (9)-Implementation of Artificial Neural Networks