The artificial neural network is divided into two stages:
1: Receive signals from other n neurons, these input signals are processed with corresponding weights
The weighted sum is passed to the next stage. (pre-activation phase)
2: Pass the pre-activated weighted result to the activation function
sum : weighted
f: activation function
Neurons
The neuron is the most basic unit of the neural network. It originally originated from the human body, imitating the neurons of the human body, and its function is also consistent with the neurons of the human body. After receiving the input of the signal, after data processing, it gives a result as an output or as the input of the next neuron.
•
Input: is the feature vector. The eigenvectors represent the direction of change. In other words, it is the method that best represents the characteristics of this thing
Towards.
•
Weight (weight): It is the feature value. Can be positive or negative, reinforcing or suppressing, just like eigenvalues. The absolute value of the weight, on behalf of
The magnitude of the influence of the input signal on the neuron is shown.
-----------------------------------------------------------------------------------------------------------------------------
The easiest way to separate these two sets of eigenvectors?
ax_+by+c=0 ->y = kx+b ->y=wx+b
Extend the above formula to n-dimensional space:
h = a1x1+a2x2+...+anxn+a0 = 0
A neuron is a model that outputs 1 when h is greater than 0, and outputs 0 when h is less than 0. Its essence is to cut the feature space into two halves, and think that the two
halves belong to two classes.
-----------------------------------------------------------------------------------------------------------------------------
Disadvantages of neurons: only one size fits all
Workaround -> Multilayer Neural Network
-----------------------------------------------------------------------------------------------------------------------------
Neural Networks
• A neural network is an operational model consisting of a large number of nodes ( neurons ) and connections between them.
• Each connection between two nodes represents a weighted value for the signal passing through the connection, called weight , which is equivalent to the memory of the artificial neural network.
• The output of the network is different according to the connection method of the network, the weight value and the activation function. The network itself is usually an approximation to a certain algorithm or function in nature, or it may be an expression of a logical strategy.
• Single layer neural network (perceptron)
-----------------------------------------------------------------------------------------------------------------------------
multilayer neural network
•
A neural network is composed of multiple neurons. The result of the previous neuron is used as the input of the next neuron, which is combined sequentially. The neural network is generally divided into three layers. The first layer is used as the input layer, the last layer is used as the output layer, and all the middle layers are hidden layers.
•
Theory proves that any multi-layer network can be approximated by a three-layer network.
•
It is generally determined by experience how many nodes the hidden layer should have, and the number of nodes can be adjusted continuously during the test to achieve the best results.
-----------------------------------------------------------------------------------------------------------------------------
feedforward neural network
•
The artificial neural network model mainly considers the topological structure of network links, neuron characteristics, learning rules, etc.
•
Among them, the feedforward neural network is also called
a multi-layer perceptron
-----------------------------------------------------------------------------------------------------------------------------
activation function
•
Activation function is a core unit of neural network design.
•
In the neural network, the neurons in the active state are called the active state, and the neurons in the inactive state are called the inhibited state. Activation functions give neurons the ability to learn and adapt by themselves.
•
The role of the activation function is to introduce nonlinear learning and processing capabilities in the neural network.
•
Commonly used activation functions (satisfy 1 nonlinearity 2 differentiability 3 monotonicity)
1 sigmoid function
2 tanh function 
3 ReLU function f(x) = max (0, x)
-----------------------------------------------------------------------------------------------------------------------------
There are two main disadvantages of sigmoid:
1. The gradient is saturated, as can be seen from the figure, the gradient of the values on both sides is 0;
2. The average value of the result is not 0, which we do not want, because this will cause the input of the neurons in the back layer to be a
signal with a non-zero mean value, which will affect the gradient
-----------------------------------------------------------------------------------------------------------------------------
Activation function
- tanh
The picture is wrong, see the next picture
-----------------------------------------------------------------------------------------------------------------------------
Activation function
-
linear rectification layer
RELU
The ReLU function is actually a piecewise linear function, which changes all negative values to 0 and keeps positive values unchanged. This operation
Known as
unilateral inhibition
.
It is precisely because of this unilateral inhibition that the neurons in the neural network also have sparse activation.
When
the model adds N layers, theoretically the activation rate of ReLU neurons will be reduced by 2 to the Nth power
.
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------
•
The operation of any algorithm must rely on a specific data structure, and the data structure used to uniformly encapsulate various data and input it into the network model is called tensor, that is, tensor. Tensors exist in different forms in different situations.
•
A major feature of tensors is dimensionality, and a 0-dimensional tensor is a constant. In Python, the dimensions of a tensor can be obtained by reading its ndim attribute. (Our commonly used arrays are equivalent to one-dimensional tensors, and a two-dimensional array is a two-dimensional tensor)
•
The so-called n-dimensional tensor is actually a one-dimensional array, and each element in the array is an n-1-dimensional tensor. It can be seen that a 3-dimensional tensor is actually a one-dimensional array, and each element in the array is a two-dimensional array.
-----------------------------------------------------------------------------------------------------------------------------
design neural network
1. Before using the neural network training data, the number of layers of the neural network and the number of units in each layer must be determined
2. When the feature vector is passed into the input layer, it is usually normalized to between 0-1 (in order to speed up the learning process)
3. Discrete variables can be encoded as each input unit corresponds to a value that may be assigned to a eigenvalue
For example: the characteristic value A may take three values (a0, a1, a2), and 3 input units can be used to represent A.
If A=a0, then the cell value representing a0 is 1, and the others are 0; 1, 0, 0
If A=a1, then the cell value representing a1 is 1, the others are 0, and so on 0,1,0
4. Neural networks can be used for both classification and regression problems
(1) For classification problems, if it is 2 categories, it can be represented by an output unit (0 and 1 represent 2 categories respectively); if there are more than 2 categories, each
One category is represented by one output unit 1 0 0 01 0
(2) There is no clear rule to design the best number of hidden layers, which can be tested and improved according to experimental tests, errors and accuracy.
-----------------------------------------------------------------------------------------------------------------------------
Deep Neural Networks
&
Deep Learning
•
The traditional neural network has developed to the case of multiple hidden layers,
•
A neural network with multiple hidden layers is called a deep neural network, and machine learning research based on deep neural networks is called
It's called deep learning.
•
If it is necessary to refine and distinguish the difference, then the deep neural network can be understood as a structure,
method optimization.
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------