Deep learning method - a brief overview of NLLloss

Many articles on the Internet are very detailed, but they also introduce various parameters by the way. If you just want to understand its function, it will be too much effort to look at it, so I will briefly summarize it with a formula:

It is not difficult to see that NLLloss+log+softmax is CrossEntropyLoss ( softmax version of the cross-entropy loss function ), and NLLloss is the last step in the cross-entropy loss function : the negative sum of the prediction results.

And it also saves you a OneHot code by the way, because it directly takes out the subscript position corresponding to the target value of each sample in the log(softmax(input)) matrix (this position is 1 in onehot, and the rest The position is 0 in onehot) and the prediction results are negatively summed.

The description of pure text will lead to abstraction, so the last two pieces of code are to illustrate:

import torch
from torch import nn

# NLLLoss+LogSoftmax
# logsoftmax=log(softmax(x))
m = nn.LogSoftmax(dim=1) #横向计算
loss = nn.NLLLoss()
torch.manual_seed(2)
# 3行5列的输入，即3个样本各包含5个特征，每个样本通过softmax产生5个输出
input = torch.randn(3, 5, requires_grad=True)
target = torch.tensor([1, 0, 4])
# NLL将取输出矩阵中第0行的第1列、第1行的第0列、第2行的第4列加负号求和
output = loss(m(input), target)
output

tensor(2.1280, grad_fn=<NllLossBackward0>)

import torch
from torch import nn

# CrossEntropyLoss交叉熵损失
# 等价于NLLloss+log+softmax
loss = nn.CrossEntropyLoss()
torch.manual_seed(2)
input = torch.randn(3, 5, requires_grad=True)
target = torch.tensor([1, 0, 4])
output = loss(input, target)
output

tensor(2.1280, grad_fn=<NllLossBackward0>)

It can be seen that the cross-entropy loss function is consistent with the result of the logsoftmax+NLL loss function , and we can use this to understand   the underlying implementation of CrossEntropyLoss .