Keras add custom Losses

Preface

Sometimes, we need to add some custom loss to the model.
These custom losses are mainly divided into two categories:

1. loss is the correct label yreal y_{real} based on the sample${and}_{real}$And predicted logit ${and}_{pred}$Calculated. This loss is mainly focused on the improvement of classification loss, such as Focal Loss and so on.
2. Loss requires the middle layer information of the model to participate in the calculation. This kind of loss is common in the regularization term for weights, and at a higher level, there is also the confrontation learning regularization term against FGSM in machine learning.

Among them, the first type of loss can be through a custom loss function, or a loss class, and then specify the loss when the model compiles.
The second type of loss cannot be specified by the model compile() function, but needs add_loss()to be specified by calling the function.

The first type of loss must be able to pass ${and}_{real}$和${and}_{pred}$, And some additional hyperparameters are calculated.
The input of this function is two parameters ${and}_{real}$和${and}_{pred}$, The output is a tensor, and the tensor must output a value for each sample.
After the definition, specify it in the model's compile.

E.g:

alpha1=0.2
alpha2=0.5
def custom_loss_function(y_true, y_pred):
   squared_difference = tf.square(y_true - y_pred)
   ##如果计算过程需要用上超参数，则先在全局作用域定义超参数，然后在函数里面直接使用这些超参数。
   return tf.reduce_mean(squared_difference, axis=-1)

model.compile(optimizer='adam', loss=custom_loss_function)

Another example is the definition of Focal Loss:

where $\gamma$ is a hyperparameter used to reconcile loss. We can define it as a global parameter when implementing it, and then use it in the function.

The second type of loss requires the neural network itself to participate in the calculation, such as L2 regularization.
The definition of this type of loss is generally after the network is built, because the calculation of these losses can easily access the output or weight of the intermediate layer.

This kind of loss does not have the same input and output convention as the first type of loss. Loss is generally an arbitrary calculation graph encapsulated with a certain function.
After the loss is defined, add add_lossthis part of the calculation graph to the model by calling the function of the model.

The relationship between the loss specified in compile and the loss specified by add_loss: The
following code can be seen in the source code of the model.compile function of keras:

        with K.name_scope('loss'):
            for i in range(len(self.outputs)):
                if i in skip_target_indices:
                    continue
                y_true = self.targets[i]
                y_pred = self.outputs[i]
                weighted_loss = weighted_losses[i]
                sample_weight = sample_weights[i]
                mask = masks[i]
                loss_weight = loss_weights_list[i]
                with K.name_scope(self.output_names[i] + '_loss'):
                    output_loss = weighted_loss(y_true, y_pred,
                                                sample_weight, mask)
                if len(self.outputs) > 1:
                    self.metrics_tensors.append(output_loss)
                    self.metrics_names.append(self.output_names[i] + '_loss')
                if total_loss is None:
                    total_loss = loss_weight * output_loss
                else:
                    total_loss += loss_weight * output_loss
            if total_loss is None:
                if not self.losses:
                    raise ValueError('The model cannot be compiled '
                                     'because it has no loss to optimize.')
                else:
                    total_loss = 0.

            # Add regularization penalties
            # and other layer-specific losses.
            for loss_tensor in self.losses:
                total_loss += loss_tensor

The previous is to calculate the loss specified in compile, and then self.lossesadd up the loss.
It self.lossesis add_lossthe loss added by the model (including all layers it uses) through the function.