Loss related summary of loss function (fine version)

table of Contents

The role of Loss loss function

Comparison of loss function loss and accuracy

The loss in the regression task mainly includes

---

The role of Loss loss function

• Loss function used to evaluate the model predicted value and the true value of the level is not the same , the depth of learning and training time is calculated model loss function, update the model parameters to reduce the error optimization, until the loss function value or decreased to achieve the target Number of training sessions.
• The loss function used by different models is generally different. The better the loss function is set, the better the performance of the model.
• The loss function can be customized

Comparison of loss function loss and accuracy

• In classification problems , accuracy is more intuitive and interpretable; for regression problems , accuracy is not available, only loss
• The accuracy rate is not differentiable and cannot be directly used for network training, and the backpropagation algorithm requires the loss function to be differentiable
• The loss function loss can be differentiated , the gradient can be calculated, and the parameters can be updated using back propagation.

The loss in the regression task mainly includes

• Mean square error (MSE): MSE represents the sum of the squares of the difference between the predicted value and the target value and then average

• L2 norm: L2 loss represents the sum of the squares of the difference between the predicted value and the target value and then squares it. L2 represents the Euclidean distance.

• The curve trend of MSE and L2 are the same. The difference is that one is seeking the average np.mean(), and the other is seeking a more square np.sqrt()

Tensorflow and keras code reflect:

#tensorflow

tf.losses.mean_squared_error(
    labels,
    predictions,
    weights=1.0,
    scope=None,
    loss_collection=tf.GraphKeys.LOSSES,
    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS
)
tf.metrics.mean_squared_error(
    labels,
    predictions,
    weights=None,
    metrics_collections=None,
    updates_collections=None,
    name=None
)


#keras

mean_squared_error(y_true, y_pred)

• Meanabsolute error (MAE): MAE represents the absolute value of the difference between the predicted value and the target value and then average

• L1 norm: L1 represents the absolute value of the difference between the predicted value and the target value, L1 is also called Manhattan distance

The difference between MAE and L1 is that one obtains the mean value np.mean(), and the other does not calculate np.sum(). The curve trends of the two are also exactly the same.

• Tensorflow and keras code reflect:

#tensorflow
tf.metrics.mean_absolute_error(
    labels,
    predictions,
    weights=None,
    metrics_collections=None,
    updates_collections=None,
    name=None
)

#keras
mean_absolute_error(y_true, y_pred)

• MSE, MAE comparison:

MAE loss is more robust to outliers, but its derivative discontinuity makes the process of finding the optimal solution inefficient; MSE loss is sensitive to outliers, but it is more stable and accurate in the optimization process.

• Other available loss function reference:

https://zhuanlan.zhihu.com/p/58883095

Supplementary knowledge

Mean absolute error MAE ( mean absolute error) and root mean square error RMSE ( root mean squared error) are the two most commonly used indicators to measure the accuracy of variables, and they are also two important scales for evaluating models in machine learning.

1. Definition

The average absolute error MAE( mean absolute error) is the average of absolute errors, which is actually a more general form of the average of errors.

or

The root mean square error RMSE ( root mean squared error), also known as the data RMSD, can also measure the average size of the error. It is the square root of the average of the squared difference between the predicted value and the actual observation.

or

 

2. Comparison

The root mean square error RMSE is more affected by outliers.

3. Application

When we are dealing with larger data sets, we cannot check every value to see if there is one or some outliers, or if all errors are systematically higher

Solution: Looking at the ratio of MAE to RMSE can help us understand whether there are large but uncommon errors.

 

---

Reference materials:

[1] https://zhuanlan.zhihu.com/p/58883095

【2】https://blog.csdn.net/u014421797/article/details/104689384

【3】https://blog.csdn.net/qq_14845119/article/details/80787753?utm_medium=distribute.pc_relevant.none-task-blog-utm_term-2&spm=1001.2101.3001.4242

【4】https://blog.csdn.net/nanhuaibeian/article/details/102746602