[Loss function] Cross Entropy Loss Cross entropy loss

1 Introduction

Several loss functions introduced on the home page are all loss functions suitable for regression problems. For classification problems, the most commonly used loss function is the cross entropy loss function Cross Entropy Loss. It is used to measure the difference between two probability distributions and is often used to evaluate the performance of classification models.

2, official

For binary classification problems, the general form of cross-entropy loss is:

$Binary Cross Entropy Loss=-\frac{1}{N} \sum_{i=1}^N\left[y_i \log \left(\hat{y}_i\right)+\left(1-y_i\right) \log \left(1-\hat{y}_i\right)\right]$

where ${N}$ is the number of samples, ${y}_i$ is the actual label, and $\hat{y}_i$ is the predicted probability of the model.

For multi-classification problems, the general form of cross-entropy loss is:

$Categorical Cross Entropy Loss=-\frac{1}{N} \sum_{i=1}^N \sum_{j=1}^C y_{i j} \log \left(\hat{y}_{i j}\right)$

Among them, ${N}$ is the number of samples, ${C}$ is the number of categories, $\hat{y}_{i j}$ is the one-hot encoding of the actual label, and ${y}_{i j}$ is ${j}$ the predicted probability of the category by the model.

3. Image

The picture above is an image of Cross Entropy Loss. The figure shows the relationship between the probability and loss of two categories (true category and wrong category). It can be seen that the loss is smaller when it is close to the target value, and the loss increases exponentially as the error becomes worse.

4. Examples

Suppose we have the following situation: we are training a model to classify three types of instances, and there are 100 samples to be tested.

As the loss function we use CrossEntropyLoss :

import torch
import torch.nn as nn

# 示例数据
torch.manual_seed(42)
num_classes = 3
num_samples = 100
y_true = torch.randint(0, num_classes, (num_samples,))
y_pred_logits = torch.randn(num_samples, num_classes)

# 定义交叉熵损失函数
criterion = nn.CrossEntropyLoss()

# 计算损失
loss = criterion(y_pred_logits, y_true)

print(f'Cross Entropy Loss: {loss.item()}')

In this case, y_pred_logitsis the output of the model, which contains the unnormalized predictions for each class. y_trueis the actual label. By passing these two to CrossEntropyLoss, the cross-entropy loss can be calculated. In actual training, you may need to incorporate an optimizer to update the model's weights to reduce loss.

5, reference

Overview of commonly used loss functions in deep learning: basic forms, principles, and characteristics (qq.com)