1.class torch.nn.L1Loss(size_average=None, reduce=None)
Function:
Computes the absolute value of the difference between
output
and
target
, optionally returning
a tensor of the same dimension
or a scalar.
Calculation formula:
parameter:
reduce (bool) -
whether the return value is a scalar, the default is
True
size_average (bool) - Effective
when
reduce=True
. When it is
True
, the returned
loss
is the average value; when it is
False
When , the sum of the loss
of each sample returned
.
code:
# ----------------------------------- L1 Loss
# 生成网络输出 以及 目标输出
output = torch.ones(2, 2, requires_grad=True)*0.5
target = torch.ones(2, 2)
# 设置三种不同参数的L1Loss
reduce_False = nn.L1Loss(size_average=True, reduce=False)
size_average_True = nn.L1Loss(size_average=True, reduce=True)
size_average_False = nn.L1Loss(size_average=False, reduce=True)
o_0 = reduce_False(output, target)
o_1 = size_average_True(output, target)
o_2 = size_average_False(output, target)
print('\nreduce=False, 输出同维度的loss:\n{}\n'.format(o_0))
print('size_average=True，\t求平均:\t{}'.format(o_1))
print('size_average=False，\t求和:\t{}'.format(o_2))output:
2.classtorch.nn.MSELoss(size_average=None,reduce=None)
Function:
Calculates the square of the difference between output and target , optionally returning a tensor of the same dimension or a scalar.
Calculation formula:
parameter:
reduce (bool) - Whether the return value is a scalar, the default is True
size_average (bool) - Effective
when
reduce=True
. When it is
True
, the returned
loss
is the average value; when it is
False
When , the sum of the loss
of each sample returned
.
code:
# ----------------------------------- MSE loss
# 生成网络输出 以及 目标输出
output = torch.ones(2, 2, requires_grad=True) * 0.5
target = torch.ones(2, 2)
# 设置三种不同参数的L1Loss
reduce_False = nn.MSELoss(size_average=True, reduce=False)
size_average_True = nn.MSELoss(size_average=True, reduce=True)
size_average_False = nn.MSELoss(size_average=False, reduce=True)
o_0 = reduce_False(output, target)
o_1 = size_average_True(output, target)
o_2 = size_average_False(output, target)
print('\nreduce=False, 输出同维度的loss:\n{}\n'.format(o_0))
print('size_average=True，\t求平均:\t{}'.format(o_1))
print('size_average=False，\t求和:\t{}'.format(o_2))output:
3.class torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=100, reduce=None, reduction='elementwise_mean')
Function:
In multi-classification tasks, the softmax activation function + cross-entropy loss function is often used, because the cross-entropy describes the difference between two probability distributions, but the output of the neural network is a vector, not in the form of a probability distribution. Therefore, the softmax activation function is needed to " normalize " a vector into the form of a probability distribution, and then use the cross-entropy loss function to calculate the loss .
Calculation formula:
parameter:
weight(Tensor)- Set the weight for each category of loss , which is often used for category imbalance problems. weight must be float
The type of
tensor
must have the same length as category
C
, that is, each category must have
a weight
.
reduce (bool) -
whether the return value is a scalar, the default is
True
size_average (bool) - Effective
when
reduce=True
. When it is
True
, the returned
loss
is the average value; when it is
False
When , the sum of the loss
of each sample returned
.
ignore_index(int)- Ignore a certain category, do not calculate its loss
, its
loss
will be
0
, and, when using
When size_average ,
the loss
of that type will not be calculated
, and the denominator when dividing will not count the samples of that type
Code and output:
# ----------------------------------- CrossEntropyLoss
loss_f = nn.CrossEntropyLoss(weight=None, size_average=True, reduce=False)
# 生成网络输出 以及 目标输出
output = torch.ones(2, 3, requires_grad=True) * 0.5 # 假设一个三分类任务，batchsize=2，假设每个神经元输出都为0.5
target = torch.from_numpy(np.array([0, 1])).type(torch.LongTensor)
print(output)
print(target)
loss = loss_f(output, target)
print('--------------------------------------------------- CrossEntropy loss: base')
print('loss: ', loss)
print('由于reduce=False，所以可以看到每一个样本的loss，输出为[1.0986, 1.0986]')
# 熟悉计算公式，手动计算第一个样本
output = output[0].detach().numpy()
output_1 = output[0] # 第一个样本的输出值
target_1 = target[0].numpy()
print("第一个样本的输出值",output_1)
print("第一个样本的类别",target_1)
# 第一项
x_class = output[target_1]
print("正确类别的概率",x_class)
# 第二项
exp = math.e
sigma_exp_x = pow(exp, output[0]) + pow(exp, output[1]) + pow(exp, output[2])
print("分母",sigma_exp_x)
#法1 ：两项相加
log_sigma_exp_x = math.log(sigma_exp_x)
loss_1 = -x_class + log_sigma_exp_x
print('--------------------------------------------------- 法1手动计算')
print('第一个样本的loss：', loss_1)
#法2：两项相除
exp_x_class = pow(exp,x_class)
loss_1 = -math.log(exp_x_class/sigma_exp_x)
print('--------------------------------------------------- 法2手动计算')
print('第一个样本的loss：', loss_1)
# ----------------------------------- CrossEntropy loss: weight
weight = torch.from_numpy(np.array([0.6, 0.2, 0.2])).float()
loss_f = nn.CrossEntropyLoss(weight=weight, size_average=True, reduce=False)
output = torch.ones(2, 3, requires_grad=True) * 0.5 # 假设一个三分类任务，batchsize为2个，假设每个神经元输出都为0.5
target = torch.from_numpy(np.array([0, 1])).type(torch.LongTensor)
loss = loss_f(output, target)
print('\n\n--------------------------------------------------- CrossEntropy loss: weight')
print('loss: ', loss) #
print('原始loss值为1.0986, 第一个样本是第0类，weight=0.6,所以输出为1.0986*0.6 =', 1.0986*0.6)
4.class torch.nn.NLLLoss(weight=None, size_average=None, ignore_index=100, reduce=None, reduction='elementwise_mean')
Function:
For example, three classification tasks, input=[-1.233, 2.657, 0.534]
, the real label is
2
(
class=2
), then
the loss
is
-0.534
. It is the output on the corresponding category, take a negative sign!
parameter:
weight(Tensor) -Sets the weight
for each category of
loss
, which is often used for category imbalance problems.
weight
must be
float
The type of
tensor
must have the same length as category
C
, that is, each category must have
a weight
.
size_average (bool) - Effective
when
reduce=True
. When
True
, the returned
loss
is the average divided by the sum of weights
Value; when it is
False , the sum of
the loss
of each sample returned
.
reduce (bool) -
whether the return value is a scalar, defaults to
True
.
ignore_index(int)
-Ignore a certain category, do not calculate its
loss
, its
loss
will be
0
, and, when using
When size_average is used,
the loss
of that type will not be calculated
, and the denominator when dividing will not count samples of that type.
Code and output:
# ----------------------------------- log likelihood loss
# 各类别权重
weight = torch.from_numpy(np.array([0.6, 0.2, 0.2])).float()
# 生成网络输出 以及 目标输出
output = torch.from_numpy(np.array([[0.7, 0.2, 0.1], [0.4, 1.2, 0.4]])).float()
output.requires_grad = True
target = torch.from_numpy(np.array([0, 0])).type(torch.LongTensor)
loss_f = nn.NLLLoss(weight=weight, size_average=True, reduce=False)
loss = loss_f(output, target)
print('\nloss: \n', loss)
print('\n第一个样本是0类，loss = -(0.6*0.7)={}'.format(loss[0]))
print('\n第二个样本是0类，loss = -(0.6*0.4)={}'.format(loss[1]))