[Deep Learning Experiment] Feedforward Neural Network (8): Model Evaluation (Customized Accuracy class that supports batch evaluation)

Table of contents

1. Experiment introduction

 2. Experimental environment

1. Configure the virtual environment

2. Library version introduction

3. Experimental content

 0. Import necessary toolkits

1. __init__(constructor)

2. update function (update evaluation indicators)

5. accumulate (calculate accuracy)

4. reset (reset evaluation indicators)

5. Construct data for testing

6. Code integration

---

1. Experiment introduction

       This article will implement an auxiliary function-calculate the accuracy of prediction. Accuracy supports the evaluation of each batch of data in each round and accumulates the results to finally obtain the evaluation results of the entire batch of data.

• updateIteratively call methods to update evaluation metrics during training or validation ;
• Use accumulatemethod to obtain cumulative accuracy;
• Methods are used resetto reset the evaluation indicators for the next round of calculations.

 2. Experimental environment

    This series of experiments uses the PyTorch deep learning framework. The relevant operations are as follows:

1. Configure the virtual environment

conda create -n DL python=3.7 

conda activate DL

pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

conda install matplotlib

 conda install scikit-learn

2. Library version introduction

software package	This experimental version	The latest version currently
matplotlib	3.5.3	3.8.0
numpy	1.21.6	1.26.0
python	3.7.16
scikit-learn	0.22.1	1.3.0
torch	1.8.1+cu102	2.0.1
torchaudio	0.8.1	2.0.2
torchvision	0.9.1+cu102	0.15.2

3. Experimental content

ChatGPT：

        Feedforward Neural Network is a common artificial neural network model, also known as Multilayer Perceptron (MLP). It is a model based on forward propagation and is mainly used to solve classification and regression problems.

        Feedforward neural network consists of multiple layers, including input layer, hidden layer and output layer. Its name "feedforward" comes from the fact that signals can only flow forward in the network, that is, from the input layer through the hidden layer and finally to the output layer, without feedback connections.

Here's how feedforward neural networks generally work:

1. Input layer: receives raw data or feature vectors as input to the network, and each input is represented as a neuron of the network. Each neuron weights the input and transforms it through an activation function to produce an output signal.

2. Hidden layer: A feedforward neural network can contain one or more hidden layers, each consisting of multiple neurons. The neurons in the hidden layer receive input from the previous layer and pass the weighted sum of the signal transformed by the activation function to the next layer.

3. Output layer: The output of the last hidden layer is passed to the output layer, which usually consists of one or more neurons. The neurons in the output layer use appropriate activation functions (such as Sigmoid, Softmax, etc.) according to the type of problem to be solved (classification or regression) to output the final result.

4. Forward propagation: The process of transmitting signals from the input layer through the hidden layer to the output layer is called forward propagation. During forward propagation, each neuron multiplies the output of the previous layer by the corresponding weight and passes the result to the next layer. Such calculations are performed layer by layer through each layer in the network until the final output is produced.

5. Loss function and training: The training process of a feedforward neural network usually involves defining a loss function that measures the difference between the model's predicted output and the true label. Common loss functions include Mean Squared Error and Cross-Entropy. By using backpropagation and optimization algorithms (such as gradient descent), the network adjusts parameters according to the gradient of the loss function to minimize the value of the loss function.

        The advantages of feedforward neural networks include the ability to handle complex nonlinear relationships, their suitability for a variety of problem types, and their ability to automatically learn feature representations through training. However, it also has some challenges, such as easy over-fitting and difficulty in processing large-scale data and high-dimensional data. In order to cope with these challenges, some improved network structures and training techniques have been proposed, such as Convolutional Neural Networks and Recurrent Neural Networks.

This series is experimental content and does not explain theoretical knowledge in detail.

(Ahem, I actually don’t have time to sort it out. I’ll come back and fill in the gaps when I have the opportunity)

​​

 0. Import necessary toolkits

import torch
from sklearn.datasets import load_iris
from torch.utils.data import Dataset, DataLoader

• Datasetand DataLoaderclasses for handling datasets and data loading

This code defines a Accuracyclass called to support model evaluation in batches, specifically calculating accuracy in classification tasks.

1. __init__(Constructor)

class Accuracy:
    def __init__(self, is_logist=True):
        self.num_correct = 0
        self.num_count = 0
        self.is_logist = is_logist

• The constructor Accuracyis called when an object is created. It accepts an optional parameter is_logist, which defaults to True, indicating whether to logistpredict the value of the form.
• self.num_correctUsed to record the number of correctly predicted samples.
• self.num_countUsed to record the total number of samples.
• self.is_logistIndicates whether logistthe predicted value is a form.

2. update function (update evaluation indicators)

def update(self, outputs, labels):
    if outputs.shape[1] == 1:
        outputs = outputs.squeeze(-1)
        if self.is_logist:
            preds = (outputs >= 0).long()
        else:
            preds = (outputs >= 0.5).long()
    else:
        preds = torch.argmax(outputs, dim=1).long()
        
    labels = labels.squeeze(-1)
    batch_correct = (preds==labels).float().sum()
    batch_count = len(labels)
    self.num_correct += batch_correct
    self.num_count += batch_count

• updateMethod is used to update the evaluation index. It accepts two parameters outputsand labels, which represent the model's predicted output and the true label respectively.
• Determine the task type based on outputsits shape.
  •  If outputsit is a two-dimensional tensor and the second dimension is 1, it represents a binary classification task.
    •   If is_logist=True, then outputsthe predicted value is converted by threshold (0) predsand converted to an integer type.
    •   If is_logist=False, then outputsthe predicted value is converted by threshold (0.5) predsand converted to integer type.
  •  If outputsit is a two-dimensional tensor and the second dimension is greater than 1, it indicates a multi-classification task. At this time, outputsthe category with the highest probability is used as the predicted value preds.
• labelsRedundant dimensions will be removed and the number of correctly predicted samples in this batch of data will be calculated batch_correct.
• Get the number of samples in this batch of data batch_count.
• Update the sum num_correctand num_countcumulatively calculate the number of correct samples and the total number of samples.

5. accumulate (calculate accuracy)

def accumulate(self):
    if self.num_count == 0:
        return 0
    return self.num_correct / self.num_count

• accumulateMethod used to calculate accuracy.
  •  If num_countit is 0, it means no update has been performed, and 0 is returned.
  • Otherwise, return the ratio of the number of correct samples divided by the total number of samples, that is, the accuracy rate .

4. reset (reset evaluation indicators)

def reset(self):
    self.num_correct = 0
    self.num_count = 0

• resetThe method is used to reset the evaluation index and num_correctreset num_countthe sum to 0 for the next round of evaluation .

5. Construct data for testing

y = torch.tensor([0, 2])
y_hat = torch.tensor([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])
acc = Accuracy()
acc.update(y_hat, y)
acc.num_correct

6. Code integration

import torch


# 支持分批进行模型评价的 Accuracy 类
class Accuracy:
    def __init__(self, is_logist=True):
        # 正确样本个数
        self.num_correct = 0
        # 样本总数
        self.num_count = 0
        self.is_logist = is_logist

    def update(self, outputs, labels):
        # 判断是否为二分类任务
        if outputs.shape[1] == 1:
            outputs = outputs.squeeze(-1)
            # 判断是否是logit形式的预测值
            if self.is_logist:
                preds = (outputs >= 0).long()
            else:
                preds = (outputs >= 0.5).long()
        else:
            # 多分类任务时，计算最大元素索引作为类别
            preds = torch.argmax(outputs, dim=1).long()

        # 获取本批数据中预测正确的样本个数
        labels = labels.squeeze(-1)
        batch_correct = (preds == labels).float().sum()
        batch_count = len(labels)
        # 更新
        self.num_correct += batch_correct
        self.num_count += batch_count

    def accumulate(self):
        # 使用累计的数据，计算总的评价指标
        if self.num_count == 0:
            return 0
        return self.num_correct / self.num_count

    def reset(self):
        self.num_correct = 0
        self.num_count = 0


y = torch.tensor([0, 2])
y_hat = torch.tensor([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])
acc = Accuracy()
acc.update(y_hat, y)
acc.num_correct