How to implement a neural network from scratch

Ready to work

神经元Is 神经网络an indispensable part. Before implementing a neural network, we need to know what a neuron is

• Neuron is the basic unit of neural network. The neuron first gets the input, and then after a certain mathematical operation, it produces an output.

• 

• In this neuron, there are two inputs and one output, and a total of three operations are performed on the two inputs:
  1. First, multiply the two inputs with weights:
  x1—>x1*w1
  x2—>x2* w2
  2. Add the results obtained, plus the bias (bias) b
  y=x1*w1 + x2*w2 + b
  3. Finally, after 激活函数processing, the output
  y=f(x1*w1 + x2*w2 + b )

• The role of the activation function is to transform unlimited inputs into predictable outputs. A commonly used function is the sigmoid function.
  The value range of the sigmoid function is from 0 to 1, that is, any input from -∞ to +∞ can become a number between 0-1 after passing through the sigmoid function.

• Based on the above neuron, we can simply implement it with code

# 定义sigmoid函数:f(x)=1/(1+e^(-x))
def sigmoid(x):
    import numpy as np
    return 1/(1+np.exp(-x))

# 定义一个神经元的类
class Neuron:
    def __init__(self, weights, bias):  #传入权重和偏置
        self.weights = weights
        self.bias = bias
    
    def feedforward(self, inputs):     
        total = np.dot(self.weights, inputs)  #数学运算
        return sigmoid(total)   # 返回经过激活函数之后的结果

test

import numpy as np

weights = np.array([1,0]) 
bias = 2

n = Neuron(weights,bias)   #实例化一个类

test = np.array([3,2])    #创建一个输入

print(n.feedforward(test))   #输出为0.9525741268224334

Build a neural network

A neural network connects neurons and uses the output of one neuron as the input of the next neuron.
We assume that the weights and biases of all neurons in this neural network are [0,1], 1, and the activation functions are all sigmoid functions. We use code to express this neural network.

class NeuronNetwork:
    def __init__(self):
        weights = np.array([0,1])
        bias = 1
        self.h1 = Neuron(weights,bias)   #h1神经元
        self.h2 = Neuron(weights,bias)   #h2神经元
        self.o1 = Neuron(weights,bias)   #o1神经元
        
    def feedforward(self,x):
        h1 = self.h1.feedforward(x)     #h1神经元的输出
        h2 = self.h2.feedforward(x)     #h2神经元的输出
            
        o1 = self.o1.feedforward([h1,h2])  #o1神经元的输出
            
        return o1

network = NeuronNetwork()

x = np.array([2,3])

print(network.feedforward(x))   #结果为：0.7216325609518421

In this way, we have successfully built a neural network, but such input and output are only for building the network and understanding his general situation, which is of no practical significance.

重点

Training a neural network

In practical applications, we need a neural network to make reasonable predictions for us, that is, to give some data and discover its internal connection, that is, to train a neural network based on these data, and when there is new data, it can be passed through the neural network. The network predicts something we need.
For example, we have a data set

Name	height	body weight	gender
a	185	81	male
b	160	48	Female
c	178	74	male
d	158	42	Female

We want to judge whether a person is a male or female based on a person’s height and weight, so we can’t arbitrarily set the weights (weights) and bias (bais), so we need to start from the data we already know and find A suitable weight and bias, so that we can predict a person's gender by height and weight.

Start

First, do a short answer to the data (height -170, weight -70, 1 means male, 0 means female)

Name	height	body weight	gender
a	5	11	1
b	-10	-22	0
c	8	4	1
d	-12	-28	0

We must have a standard to define whether this neural network is good or not, so as to improve it. This is loss.
In general, with 均方误差defined loss.

MSE=$\frac{1}{n}$$\sum _{i=1}^n({and}_t-{and}_p{)}^2$

n is the number of samples, from the above we can see that it is 4
y is the predicted output value, that is, gender, male is 1, female is 0
${and}_t$Represents the true gender, ${and}_p$Represents the predicted gender
. Obviously, the smaller the mean square error, the better the prediction result. Then our goal is to minimize the mean square error. Define it as 损失函数.

Optimize the neural network

Using the 随机梯度下降optimization algorithm to train the neural network. I.e.,
$w_1$ <— $w_1$ - η($\partial L$/$\partial$$w_1$)
η is a constant, called the learning rate, it determines how fast we train the network. This formula has a new weight $w_1$.
(The derivation of the formula is updated...) The
training process:
1. Select a sample from the data set;
2. Calculate the derivative function of the loss function to all weights and bias
3. Use the update formula to update each weight
4. Go back 1, until all data sets are looped.
This step requires a certain knowledge of seeking partial derivatives, that is, the derivative function of the loss function to all weights and biases is calculated in step 2.

import numpy as np

def sigmoid(x):
    return 1/(1+np.exp(-x))

def deriv_sigmoid(x):
    fx = sigmoid(x)
    return fx*(1-fx)

def mse_loss(y_true, y_pred):
    return ((y_true - y_pred)**2).mean()

class NeuralNetwork:
    def __init__(self):
        self.w1 = np.random.normal()
        self.w2 = np.random.normal()        
        self.w3 = np.random.normal()        
        self.w4 = np.random.normal()
        self.w5 = np.random.normal()
        self.w6 = np.random.normal()
        
        self.b1 = np.random.normal()
        self.b2 = np.random.normal()
        self.b3 = np.random.normal()
    def feedforward(self,x):
        h1 = sigmoid(self.w1*x[0] + self.w2*x[1] + self.b1)
        h2 = sigmoid(self.w3*x[0] + self.w4*x[1] + self.b2)
        o1 = sigmoid(self.w5*h1 + self.w6*h2 + self.b3)
        
        return o1
    def train(self, data, all_y_trues):
        
        learn_rate = 0.1
        epochs = 500
        for epochs in range(epochs):
            for x, y_true in zip(data,all_y_trues):
                sum_h1 = self.w1*x[0] + self.w2*x[1] + self.b1
                h1 = sigmoid(sum_h1)
            
                sum_h2 = self.w3*x[0] + self.w4*x[1] + self.b2
                h2 = sigmoid(sum_h2)
            
                sum_o1 = self.w4*h1 + self.w5*h2 + self.b3
                o1 = sigmoid(sum_o1)
                y_pred = o1
            
                d_L_d_y_pred = -2 * (y_true - y_pred)
            
                d_ypred_d_w5 = h1*deriv_sigmoid(sum_o1)
                d_ypred_d_w6 = h2*deriv_sigmoid(sum_o1)
                d_ypred_d_b3 = deriv_sigmoid(sum_o1)
            
                d_ypred_d_h1 = self.w5*deriv_sigmoid(sum_o1)
                d_ypred_d_h2 = self.w6*deriv_sigmoid(sum_o1)
            
                d_h1_d_w1 = x[0]*deriv_sigmoid(sum_h1)
                d_h1_d_w2 = x[1]*deriv_sigmoid(sum_h1)
                d_h1_d_b1 = deriv_sigmoid(sum_h1)
            
                d_h2_d_w3 = x[0]*deriv_sigmoid(sum_h2)
                d_h2_d_w4 = x[1]*deriv_sigmoid(sum_h2)
                d_h2_d_b2 = deriv_sigmoid(sum_h2)
            
                self.w1 -= learn_rate*d_L_d_y_pred*d_ypred_d_h1*d_h1_d_w1
                self.w2 -= learn_rate*d_L_d_y_pred*d_ypred_d_h1*d_h1_d_w2
                self.b1 -= learn_rate*d_L_d_y_pred*d_ypred_d_h1*d_h1_d_b1
                
                self.w3 -= learn_rate*d_L_d_y_pred*d_ypred_d_h2*d_h2_d_w3
                self.w4 -= learn_rate*d_L_d_y_pred*d_ypred_d_h2*d_h2_d_w4
                self.b2 -= learn_rate*d_L_d_y_pred*d_ypred_d_h2*d_h2_d_b2
            
                self.w5 -= learn_rate*d_L_d_y_pred*d_ypred_d_w5
                self.w6 -= learn_rate*d_L_d_y_pred*d_ypred_d_w6
                self.b3 -= learn_rate*d_L_d_y_pred*d_ypred_d_b3

Training neural network is too late to update due to time issues, it will squeeze time to update