Python handwriting neural networks (a) BP neural network

purpose

  After completing the derivation of BP neural network , I would use python (without deep learning framework!) Own handwriting a simple BP neural network, in order to deepen the understanding of the neural network, and welcome to point out the problem, to study with discuss ~

Design ideas

  The only use I numpy library, for straightforward, using the class of encapsulation, comprising a made: four-node input layer, a hidden layer of five nodes, and a node of the output layer of the neural network BP.
  This objective neural network is the sum of two numbers with the first two digits of the input layer and the compared result is output greater than 1, is smaller than 0 is output. (This objective can be said for the neural network very simple)
  I first define a class BP neural network, which includes initialization function, activation function, forward propagation function to propagate function and after. Then write a data set as a function of generated data we need in order to manufacture and its corresponding output. Finally, in the main function, we defined the data size, batch size, learning rate and the maximum number of iterations can run up it!

Code and results

Code

import numpy as np

class BP_network(object):
    #The network is like:
    #    l0 -> w1 -> l1 -> w2 -> (l2 == y?)
    def __init__(self):
        # 4 node in input layer
        # 5 node in the hidden layer
        self.w1 = 2*np.random.random((4, 5)) - 1                                     # (4, 5)
        # 1 node for output layer
        self.w2 = 2*np.random.random((5, 1)) - 1                                     # (5, 1)

    def sigmoid(self, x, deriv=False):
        if deriv == True:
            return x * (1-x)
        return 1/(1+np.exp(-x))

    def forward_prop(self, input_data):
        self.l0 = input_data
        self.l1 = self.sigmoid(np.array(np.dot(self.l0, self.w1), dtype=np.float32)) # (batch_sz , 5)
        self.l2 = self.sigmoid(np.array(np.dot(self.l1, self.w2), dtype=np.float32)) # (batch_sz , 1)
        l2_error = label[st_idx:ed_idx] - self.l2                                    # (batch_sz , 1)
        return self.l2, l2_error

    def backward_prop(self, output_error):
        l2_error = output_error
        l2_delta = l2_error * self.sigmoid(self.l2, True)                            # (batch_sz , 1)
        l1_delta = np.dot(l2_delta, self.w2.T) * self.sigmoid(self.l1, True)         # (batch_sz , 5)
        self.w2 += lr * np.array(np.dot(self.l1.T, l2_delta), dtype=np.float32)      # (5 , 1)
        self.w1 += lr * np.array(np.dot(self.l0.T, l1_delta), dtype=np.float32)      # (4 , 5)

# data  :  [x0, x1, x2, x3]
# label :  x0 + x1 - x2 - x3 > 0? 1 : 0
def generate_data(data_size):
    data = np.empty((1, 4), dtype=object);
    label = np.empty((1, 1), dtype=object)
    for i in range(0, data_size):
        x = np.array([np.random.randint(0, 10, size=4)])
        res = int((x[0][0]+x[0][1] -x[0][2]-x[0][3])>0)
        y = np.array([[res]])
        data = np.concatenate((data, x), axis=0)
        label = np.concatenate((label, y), axis=0)
    return data[1:], label[1:]

if __name__ == '__main__':
    # size of data, batch size
    data_size = 100; batch_sz = 10;
    # learning rate, max iteration
    lr = 0.1;      max_iter = 5000;
    data, label = generate_data(data_size)
    NN = BP_network()
    for iters in range(max_iter):
        # starting index and ending index for fetching data
        st_idx = (iters % 10) * batch_sz; ed_idx = st_idx + batch_sz
        input_data = data[st_idx : ed_idx]
        outcome, output_error = NN.forward_prop(input_data)
        if iters % 450 == 0:
            print "The loss is %f" % (np.mean(np.abs(output_error)))
            #print output_error.tolist()
            continue
        NN.backward_prop(output_error)

operation result

The loss is 0.585963
The loss is 0.095126
The loss is 0.071693
The loss is 0.057002
The loss is 0.047533
The loss is 0.041154
The loss is 0.036482
The loss is 0.032900
The loss is 0.030066
The loss is 0.027768
The loss is 0.025866
The loss is 0.024263
[Finished in 0.6s]

  Look, this loss in declining! Are interested, we can also print out the neural network output is compared with the actual results, found that the neural network is indeed in a continuous learning ~
  (the next section, convolution neural network to achieve handwritten with python, please stay tuned!)