Simple explanation and use of common loss functions in PyTorch

loss function

The loss function is also called the objective function. One of the two parameters required to compile the neural network model. Another essential is the optimizer, which I will explain in detail later.

Important
The loss function is the function of the direct difference between the computer label value and the predicted value.

Here we will end several common calculation methods of loss functions. Many types of predefined functions are also defined in pytorch. The specific formulas do not need to be studied in depth (you may not necessarily remember them after learning), and what you can do here is to understand for the time being.

Let's first define two two-dimensional arrays, and then use different loss functions to calculate their loss values.

import torch
from torch.autograd import Variable
import torch.nn as nn

sample=Variable(torch.ones(2,2))
a=torch.Tensor(2,2)
a[0,0]=0
a[0,1]=1
a[1,0]=2
a[1,1]=3
target=Variable(a)
print(sample,target)

Here:
the value of sample is tensor([[1., 1.],[1., 1.]])

The value of target is tensor([[0., 1.],[2., 3.]])

nn.L1Loss

$$loss(x,y)=\frac{1}{N}\sum _{i=1}^N|x-y|$$
L1Loss calculation method is very simple, take the average of the absolute error of the predicted value and the real value.

loss=FunLoss(sample,target)['L1Loss']
print(loss)

Printed out in the console is

tensor(1.)

Its calculation process is like this: $(\left|0-1\right|+|1-1|+|2-1|+\left|3-1\right|)/4=1$ , the first calculation is the sum of absolute values, and then the average.

nn.SmoothL1Loss

The error of SmoothL1Loss is the square loss on (-1, 1), and the L1 loss in other cases.
$$loss(x,y)=\frac{1}{N}\{\begin{array}{cc}\frac{1}{2}(x_i-y_i{)}^{2}if\left|x_i-y_i\right|<1& \\ |x_i-y_i|-\frac{1}{2}otherwise& \end{array}$$

loss=FunLoss(sample,target)['SmoothL1Loss']
print(loss)

Printed out in the console is

tensor(0.6250)

nn.MSELoss

Squared loss function. Its calculation formula is the average of the sum of squares between the predicted value and the true value.
$$loss(x,y)=\frac{1}{N}\sum _{i=1}^N{|x-y|}^2$$

loss=FunLoss(sample,target)['MSELoss']
print(loss)

Printed out in the console is

tensor(1.5000)

nn.CrossEntropyLoss

Cross entropy loss formula
$$\begin{gathered} loss(x,label)=-log\frac{e^{x_{label}}}{{\sum }_{j=1}^N{e}^{x_j}} \\ =-x_{label}+log\sum _{j=1}^N{e}^{x_j} \end{gathered}$$

This formula is often used in image classification neural network models.

loss=FunLoss(sample,target)['CrossEntropyLoss']
print(loss)

Printed out in the console is

tensor(2.0794)

nn.NLLLoss

Negative log-likelihood loss function
$$loss(x,label)=-x_{label}$$
It should be noted that here $x_{label}$It is different from the cross-entropy loss above, here is the value after the log operation. This loss function is generally used in image recognition models.

loss=FunLoss(sample,target)['NLLLoss']
print(loss)

Here, the console reports an error, requiring 0D or 1D target tensors, and does not support multiple targets. Some other conditions may be required, and we will talk about them here if we meet them.

Loss function modular design

class FunLoss():
    def __init__(self, sample, target):
        self.sample = sample
        self.target = target
        self.loss = {
    
    
            'L1Loss': nn.L1Loss(),
            'SmoothL1Loss': nn.SmoothL1Loss(),
            'MSELoss': nn.MSELoss(),
            'CrossEntropyLoss': nn.CrossEntropyLoss(),
            'NLLLoss': nn.NLLLoss()
        }

    def __getitem__(self, loss_type):
        if loss_type in self.loss:
            loss_func = self.loss[loss_type]
            return loss_func(self.sample, self.target)
        else:
            raise KeyError(f"Invalid loss type '{
      
      loss_type}'")

if __name__=="__main__":
    loss=FunLoss(sample,target)['NLLLoss']
    print(loss)

Summarize

This blog is for readers who want to understand common loss functions in PyTorch. Through FunLoss, we can simply call it ourselves.