Implementing a neural network (python) from zero to one: one

Perceptron algorithm

In 1957, it was proposed by American scholar Frank Rsenblatt, the origin algorithm of neural network.

concept

Receives multiple input signals and outputs one signal (0 or 1)

Here is an example of a perceptron that accepts three input signals

Before the input signal is received by the neuron, it will be multiplied by its respective weight ¹ , and then processed by the neuron (the processing here is the weighted summation of the input neurons) to obtain the pseudo output value of a neuron, when When the quasi-output value is greater than a certain threshold, the neuron will output 1, indicating that the neuron is activated ²

Express the above process in mathematical expressions as:

From the above mathematical expression, we can find that the threshold actually represents the ease with which the neuron is activated, and the actual output value final has a linear relationship with the input, so the above mathematical expression can also be written as

Implementing Simple Logic Circuits Using Perceptrons

We probably know the theory of the perceptron, how can we use it to solve practical problems?

AND gate

Features:

If the input is all 1, output 1, otherwise output 0

truth table

The AND gate is a gate circuit with two inputs and one output. The relationship between the input signal and the output signal refers to the following truth table

x1	x2	final
0	0	0
0	1	0
1	0	0
1	1	1

Perceptron implementation

def AND(x1,x2):
	w1,w2,theta=0.5,0.5,0.7
	tmp=w1*x1+w2*x2
	if tmp<=theta:
		return 0
	else:
		return 1		

Improvement:
From the above algorithm, we can see that when calculating the output tmp, we need to calculate the expression of multiplying the weight and the input value. When there are many input values, this expression will be very verbose. In order to solve this problem, we introduce numpy array operations .

def AND(x1,x2):
	x=np.array([x1,x2])		# 引入numpy数组
	w=np.arry([0.5,0.5])	# 引入numpy数组
	b=-0.7	
	tmp=np.sum(w*x)+b
	if tmp<=0:
		return 0
	else:
		return 1

NAND gate

Features

Output 0 if both inputs are 1, otherwise output 1

truth table

x1	x2	final
0	0	1
0	1	1
1	0	1
1	1	0

Perceptron implementation

def NAND(x1,x2):
	x=np.array([x1,x2])
	w=np.arry([-0.5,-0.5])
	b=0.7
	tmp=np.sum(w*x)+b
	if tmp<=0:
		return 0
	else:
		return 1	
		

OR gate

Features

As long as one output signal is 1, output 1, otherwise output 0

truth table

x1	x2	final
0	0	0
0	1	1
1	0	1
1	1	1

Perceptron implementation

def OR(x1,x2):
	x=np.array([x1,x2])
	w=np.arry([0.5,0.5])
	b=-0.2
	tmp=np.sum(w*x)+b
	if tmp<=0:
		return 0
	else:
		return 1

About the limitations of perceptrons

1. The essence of neural network training : carefully observe the implementation process of the above three logic circuits, we can see that only the weight parameters and thresholds are different, but different functions are realized. In fact, the essence of neural network training is to continuously optimize the weight parameters , and pay more attention to ³ in the future learning process

2. Perceptron limitations :
   

We see that AND gates, NAND gates, and OR gates can all be divided into two spaces by a linear function, one of which outputs 0, and the other space outputs 1.
But XOR gates cannot be divided by a straight line anyway. Two spaces can only be divided into two spaces by using a curve. The space divided by straight lines is called linear space , and the space divided by curves is called nonlinear space.

XOR gate

A single perceptron cannot represent an XOR gate, but we can do so by stacking multiple perceptrons

Perceptron implementation

There are many ways to implement XOR gates in multi-layer perceptrons. The author introduces one here: x1, x2 inputs are used as the input of NAND gate and OR gate, and the output of NAND gate and OR gate is used as the input of AND gate

Code

def XOR(x1,x2):
	return AND(NAND(x1,x2),OR(x1,x2))

---

1. Each input signal value has its own weight, and these weights represent the importance of the input signal. That is, the higher the weight value, the greater the influence of the corresponding signal value on the output value↩︎

2. In fact, in the working mechanism of biological neurons, neurons do not simply generate output values ​​for all input values, but will respond when the input values ​​reach a certain threshold. The author's understanding: neural network is a bionic knowledge, and the setting of perceptron is also inspired by biological neurons↩︎

3. The author's perception: The so-called good neural network model is that the optimized weight parameters can achieve an excellent fitting effect for a certain problem↩︎