Neural Networks
Neural network is the core of deep learning. Before understanding neural network, let's understand a few concepts:
Intelligence ( Intelligence) is the comprehensive ability of individual purposeful behavior, reasonable thinking and effective adaptation to the environment. In other
words, intelligence is the ability of individuals to understand objective things and use knowledge to solve problems.
Artificial Intelligence ( AI) was first introduced in 1956. It mainly studies how to make computers imitate the
human brain to engage in thinking activities such as reasoning, design, thinking, and learning, so as to solve and deal with more complex problems. To put it simply, artificial
intelligence is the study of how to make computers work by imitating the human brain.
Artificial Neural Network (ANN) is a brain-based intelligent
information processing system designed to imitate the structure and functions of the human brain. Usually, the human brain neural network is abstracted from the perspective of mathematical and physical methods and information processing
, and a simplified model is established. To put it simply, it is a mathematical model that can be realized by electronic circuits or
simulated by computer programs. It is a research method of artificial intelligence.
Artificial Intelligence (AI) was first introduced in 1956. It mainly studies how to make computers imitate the
human brain to engage in thinking activities such as reasoning, design, thinking, and learning, so as to solve and deal with more complex problems. Simply put,
intelligence is still the study of how to make a computer imitate the human brain to work.
Due to the different starting points, methodologies and application fields of research, there are several representative schools:
-
Symbolism School : The hypothesis put forward by Newell and Simon in 1967, that artificial intelligence originates from mathematical logic, and
intelligent behavior is described through mathematical logic, and later developed the theory of heuristic algorithm->expert system->knowledge engineering. -
Connectionism School : Representatives are McCulloch and Pitts, who believe that artificial intelligence comes from bionics, especially the study of the human brain
, and proposed the MP model, and later derived artificial neural networks based on this model. -
Behaviorism : It is believed that artificial intelligence originates from cybernetics. The cybernetics and self-organization systems proposed by Wiener et al. are based on
simulating the intelligent behavior and role of human beings in the control process, such as self-organization, self-optimization, self-adaptation, and self-organization. study etc.
human neuron
Let's take a look at how a human neuron works:
Neuron is the basic unit of brain tissue and the unit of nervous system structure and function. Different neuron forms and functions also have
differences, and their common structures are simplified as follows:
- Cell body: the main body of a neuron, composed of nucleus, cytoplasm, cell membrane, etc. The cell membrane has different permeability to different ions in the cell fluid, resulting in a difference in ion concentration, resulting in a resting potential that is negative inside and positive outside.
- Dendrites : receive input from other neurons through dendrites
- Axons : Efferent cell bodies generate outgoing electrochemical signals
- Synapse : A communication connection between neurons and the cell body or dendrites of other neurons through an axon terminal, which is equivalent to the input and output interface between neurons
Synapses are the input and output interfaces of neurons. Dendrites and cell bodies are used as input terminals to receive input signals from prominent points; cell bodies are equivalent to a processor, and each part of the dendrites and cell bodies receives signals from other neurons. The input signals of the neuron are combined and triggered under certain conditions to generate an output signal. The output signal is transmitted along the axon to the terminal, and the axon terminal is used as the output terminal to transmit the output signal to other neurons through the synapse.
The generation, transmission and processing of biological neuron information is an electrochemical activity, and its mechanism is as follows:
-
Information generation : At a given moment, neurons are always in one of three states - resting, excited and inhibited. Under external stimuli, when
the excitability of a neuron is greater than a certain threshold potential, the neuron is excited and sends out nerve impulses. -
Transmission and reception : The nerve impulse signal is transmitted along the axon to each branch at its end, and the transmission and reception are completed through the synapse. There are two types of synapses: excitatory synapses and inhibitory synapses. When the potential of excitatory synapses exceeds a certain threshold potential, the latter neuron will have a nerve pulse
output passed to the next neuron. -
Information integration : receive the pulse input from each axon, and according to the input, it can reach different parts of the neuron. Different input parts have
different weights on neurons. The potential change caused by the stimuli 3 | produced at the same time is roughly equal to the algebraic sum of the potential changes caused by individual stimuli 3 |. Neurons accumulate and integrate inputs in space and time to determine the timing and strength of outputs -
Biological neural network : A biological neural network is formed by connecting multiple biological neurons in a definite manner and topological structure. It is a more dexterous and complex biological information processing system, showing complex information processing capabilities on a macro level.
MP model
In 1943, psychologist McCulloch and mathematician w. Pitts proposed the MP model based on the characteristics of biological neurons. The model simplifies and generalizes the information processing process of biological neurons.
- Each neuron is an information processing unit with multiple input and single output
- The input of neurons is divided into two types: excitatory input and inhibitory input
- Neurons have spatial integration properties and threshold properties
- There is a fixed time delay between neuron input and output
- Neglecting temporal integration, ignoring refractory period after excitatory period
activation function
Activation Function : Also called connection function, transfer function, transformation function or activation function. It is used to simulate the connection between neuron output and its activation state: the activation state is reached after the input reaches a certain threshold, otherwise it is the inhibition state. Different activation functions will make neurons have different information processing characteristics. For the neural network, the main function of the activation function is to perform linear transformation and increase the nonlinear expression ability of the system.
sign function
sigmoid function
Tanh function
arctan function
Neural Network Classification
According to the direction of information flow, it can be divided into feedforward network and feedback network.
- Feedforward network : network information advances layer by layer from the input layer to each hidden layer and then to the output layer
- Feedback network : All nodes in the feedback network have information processing functions, and each node can receive input and output at the same time.
feedforward neural network
Feedforward neural network (FeedForward NN): It is the simplest neural network, which adopts a one-way multi-layer structure, and each neuron is arranged in layers, and each neuron is only connected to the neurons of the previous layer. Receive the output of the previous layer and output it to the next layer, there is no feedback between layers.
A feedforward network includes three types of nodes:
- Input Nodes: External information is input, no calculation is performed, and information is only passed to the next layer of nodes
- Hidden Nodes (Hidden Nodes): Receive the input of the upper layer node, perform calculations, and pass the information to the next layer node
- Output Nodes: Receive the input of the upper-layer nodes, perform calculations, and output the results
The input layer and the output layer must have, and the hidden layer can be absent, that is, a single-layer perceptron, and the hidden layer can also be more than one layer, and the feed-forward network with a hidden layer is a multi-layer perceptron
feedback neural network
Feedback neural network (FeedBack NN): Also known as recursive network and regression network, it is a neural network system that connects the output to the input layer after a one-step time shift. In this type of network, neurons can be interconnected, and the output of some neurons will be fed back to neurons in the same layer or even the previous layer. The common ones are Hopfield neural network, EIman neural network, Boltzmann machine, etc.
The main difference between feedforward neural network and feedback neural network:
- There is no connection between the neurons in each layer of the feed-forward neural network, and the neurons only accept the data from the upper layer, and pass it to the next layer after processing, and the data flows forward; the neurons in the feedback neural network layer are connected, and the data can be transmitted in the same layer. Flow between layers or feedback to the front layer
- The feed-forward neural network does not consider the time lag effect between the output and the input, but only expresses the mapping relationship between the output and the input; the feedback neural network considers the time delay between the output and the input, and needs to use dynamic equations to describe the system model.
- The learning of feed-forward neural network mainly adopts the error correction method (such as BP algorithm), the calculation process is generally slow, and the convergence speed is also relatively slow; the feedback neural network mainly uses Hebb learning rules, and the calculation convergence speed is generally very fast
- Compared with feedforward neural network, feedback neural network is more suitable for application in the fields of associative memory and optimization calculation.