Deep learning (python) - neural network (Artificial Neural Networks) activation function code

Table of contents

1 Overview

2. Activation function

(1) Basic concepts

 (2) sigmoid function (commonly used)

 code (sigmoid)

 (3) Step function

 code (step function)

(4) ReLU function (currently commonly used)

 Code (ReLU function)

---

1 Overview

        In the perceptron, the weight w is manually input, and the neural network can automatically learn the appropriate weight parameters from the data. It can be understood that the neural network grabs the characteristics of the data from the data to obtain weights, and finally uses the weights to identify unfamiliar data.

The neural network structure diagram is as follows:

 The above picture shows a 3-layer neural network, which is the simplest neural network structure, in which the middle layer is sometimes called the "hidden layer". Similar to the perceptron, in the neural network, we input data samples as the input signal, and then obtain the weight through the feature calculation of the data obtained in the hidden layer, and finally obtain the required output result in the output layer.

2. Activation function

(1) Basic concepts

First, let's look at the formula of the perceptron:

Simplify the perceptron formula to  

where h(x):

From this you can get 

a=b+w1x1+w2x2

y=h(a)

where h(a) is the activation function. Its structure is as follows:

 (2) sigmoid function (commonly used)

official:

 where exp(-x) represents the -x square of e. (This formula does not need to be remembered, it is just a calculation method)

Code (sigmoid):

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

 (3) Step function

Both the step function and the sigmoid function are nonlinear functions, but unlike sigmoid, the sigmoid function is a smooth curve, while the step function is a sharply falling or rising curve.

The function curves are compared as follows:

 code (step function)

def step_function(x):
    y = x > 0
    return y.astype(np.int)

(4) ReLU function (currently commonly used)

At present, the ReLU function is widely used, and its formula is as follows:

Code (ReLU function):

def relu(x):
    return np.maximum(0, x)