[Deep Learning Experiment] Feedforward Neural Network (3): Customized multi-layer perceptron (activation function logistic, linear layer calculation Linear)

Table of contents

1. Experiment introduction

2. Experimental environment

1. Configure the virtual environment

2. Library version introduction

3. Experimental content

0. Import necessary toolkits

1. Build a data set

2. Activation function logistic

3. Linear layer operator Linear

4. Two-layer feedforward neural network MLP

5. Model training

1. Experiment introduction

This experiment implements a simple two-layer feedforward neural network
- activation function logistic
- Linear layer operator Linear

2. Experimental environment

This series of experiments uses the PyTorch deep learning framework. The relevant operations are as follows:

1. Configure the virtual environment

conda create -n DL python=3.7

conda activate DL

pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

conda install matplotlib

 conda install scikit-learn

2. Library version introduction

software package	This experimental version	The latest version currently
matplotlib	3.5.3	3.8.0
numpy	1.21.6	1.26.0
python	3.7.16
scikit-learn	0.22.1	1.3.0
torch	1.8.1+cu102	2.0.1
torchaudio	0.8.1	2.0.2
torchvision	0.9.1+cu102	0.15.2

3. Experimental content

ChatGPT：

        Feedforward Neural Network is a common artificial neural network model, also known as Multilayer Perceptron (MLP). It is a model based on forward propagation and is mainly used to solve classification and regression problems.

        Feedforward neural network consists of multiple layers, including input layer, hidden layer and output layer. Its name "feedforward" comes from the fact that signals can only flow forward in the network, that is, from the input layer through the hidden layer and finally to the output layer, without feedback connections.

Here's how feedforward neural networks generally work:

Input layer: receives raw data or feature vectors as input to the network, and each input is represented as a neuron of the network. Each neuron weights the input and transforms it through an activation function to produce an output signal.

Hidden layer: A feedforward neural network can contain one or more hidden layers, each consisting of multiple neurons. The neurons in the hidden layer receive input from the previous layer and pass the weighted sum of the signal transformed by the activation function to the next layer.

Output layer: The output of the last hidden layer is passed to the output layer, which usually consists of one or more neurons. The neurons in the output layer use appropriate activation functions (such as Sigmoid, Softmax, etc.) according to the type of problem to be solved (classification or regression) to output the final result.

Forward propagation: The process of transmitting signals from the input layer through the hidden layer to the output layer is called forward propagation. During forward propagation, each neuron multiplies the output of the previous layer by the corresponding weight and passes the result to the next layer. Such calculations are performed layer by layer through each layer in the network until the final output is produced.

Loss function and training: The training process of a feedforward neural network usually involves defining a loss function that measures the difference between the model's predicted output and the true label. Common loss functions include Mean Squared Error and Cross-Entropy. By using backpropagation and optimization algorithms (such as gradient descent), the network adjusts parameters according to the gradient of the loss function to minimize the value of the loss function.

        The advantages of feedforward neural networks include the ability to handle complex nonlinear relationships, their suitability for a variety of problem types, and their ability to automatically learn feature representations through training. However, it also has some challenges, such as easy over-fitting and difficulty in processing large-scale data and high-dimensional data. In order to cope with these challenges, some improved network structures and training techniques have been proposed, such as Convolutional Neural Networks and Recurrent Neural Networks.

This series is experimental content and does not explain theoretical knowledge in detail.

(Ahem, I actually don’t have time to sort it out. I’ll come back and fill in the gaps when I have the opportunity)

0. Import necessary toolkits

import torch
from torch import nn

1. Build a data set

input = torch.ones((1, 10))

An input tensor `input` is created with size (1, 10).

2. Activation function`logistic`

def logistic(z):
    return 1.0 / (1.0 + torch.exp(-z))

The characteristic of the logistic function is to map the input value to an output value between 0 and 1, which can be regarded as a probability estimate. When the input value approaches positive infinity, the output value approaches 1; when the input value approaches negative infinity, the output value approaches 0. Therefore, the logistic function is often used in binary classification problems to interpret the output value as a probability value, which can be used to predict the probability that a sample belongs to a certain class. In neural networks, the introduction of logistic functions can introduce nonlinear characteristics, allowing the network to learn more complex patterns and representations.

3. Linear layer operator Linear

class Linear(nn.Module):
    def __init__(self, input_size, output_size):
        super(Linear, self).__init__()
        self.params = {}
        self.params['W'] = nn.Parameter(torch.randn(input_size, output_size, requires_grad=True))
        self.params['b'] = nn.Parameter(torch.randn(1, output_size, requires_grad=True))
        self.grads = {}
        self.inputs = None

    def forward(self, inputs):
        self.inputs = inputs
        outputs = torch.matmul(inputs, self.params['W']) + self.params['b']
        return outputs

Linearclass is a custom linear layer, inherited from nn.Module,
- It has two parameters: input_sizeand output_size, which represent the size of the input and output respectively.
During initialization, two parameters are created: Wand b, representing weights and biases respectively, both of which are trainable tensors and nn.Parameterencapsulated by .
- paramsand gradsare dictionary type attributes used to store parameters and gradients;
- inputsis a temporary variable used to store input.
forwardThe method implements the logic of forward propagation and uses input and parameters to calculate the output.

4. Two-layer feedforward neural network MLP

class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MLP, self).__init__()
        self.fc1 = Linear(input_size, hidden_size)
        self.fc2 = Linear(hidden_size, output_size)

    def forward(self, x):
        z1 = self.fc1(x)
        a1 = logistic(z1)
        z2 = self.fc2(a1)
        a2 = logistic(z2)
        return a2

Two linear layer objects are created during initialization Linear: fc1andfc2
forwardThe method realizes the forward propagation process of the entire neural network:
- The input xfirst passes through the first linear layer fc1,
- Then logisticactivate it through the function,
- Then through the second linear layer fc2,
- Finally, after another logisticfunction activation,
- and return the final output.

5. Model training

input_size, hidden_size, output_size = 10, 5, 2
net = MLP(input_size, hidden_size, output_size)
output = net(input)
print(output)

input_sizeThree variables , , hidden_sizeand , are defined output_size, representing the input size, hidden layer size and output size respectively.
MLPAn object is created net, and the input inputis passed into the model for forward calculation and the output is obtained output. Finally print the output.

6. Code integration

# 导入必要的工具包
import torch
from torch import nn


# 线性层算子，请一定注意继承自 nn. Module, 这会帮你解决许多细节上的问题
class Linear(nn.Module):
    def __init__(self, input_size, output_size):
        super(Linear, self).__init__()
        self.params = {}
        self.params['W'] = nn.Parameter(torch.randn(input_size, output_size, requires_grad=True))
        self.params['b'] = nn.Parameter(torch.randn(1, output_size, requires_grad=True))
        self.grads = {}
        self.inputs = None

    def forward(self, inputs):
        self.inputs = inputs
        outputs = torch.matmul(inputs, self.params['W']) + self.params['b']
        return outputs


# 实现一个两层的前馈神经网络
class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MLP, self).__init__()
        self.fc1 = Linear(input_size, hidden_size)
        self.fc2 = Linear(hidden_size, output_size)

    def forward(self, x):
        z1 = self.fc1(x)
        a1 = logistic(z1)
        z2 = self.fc2(a1)
        a2 = logistic(z2)
        return a2


# Logistic 函数
def logistic(z):
    return 1.0 / (1.0 + torch.exp(-z))

input = torch.ones((1, 10))
input_size, hidden_size, output_size = 10, 5, 2
net = MLP(input_size, hidden_size, output_size)
output = net(input)
print(output)