[Recommended collection] 12 regression evaluation indicators for machine learning (with Python code)

Have you ever encountered various evaluation metrics and didn’t know what they meant? Or don't know how to use Numpy for calculations?

In this article we will introduce common regression evaluation indicators, including formulas and Numpy calculation codes.

Article directory

technology improvement

This article is shared with the authorization of a friend from the fan group. If you want to join the technical exchange, the best way to add a note is: source + direction of interest, so as to find like-minded friends.

Method ①, add WeChat account: dkl88191, remarks: from CSDN+machine learning
Method ②, WeChat search public account: Python learning and data mining, background reply: add group

Mean Absolute Error，MAE

Mean Absolute Error (MAE), also known as L1 loss, is one of the simplest loss functions and an easy-to-understand evaluation metric. It is calculated by taking the absolute difference between the predicted and actual values and averaging it over the entire data set. Mathematically, it is the arithmetic mean of the absolute errors. MAE only measures the magnitude of errors, not their orientation. The lower the MAE, the more accurate the model is.

advantage:

Since absolute values are used, all errors are weighted in the same proportion.
If the training data has outliers, MAE does not penalize high errors caused by outliers.
It provides an average measure of how well the model is performing.

shortcoming:

Sometimes large errors from outliers end up being treated the same as low errors.
Not differentiable at zero. Many optimization algorithms tend to use differentiation to find the optimal values of parameters in the evaluation metric. Computing gradients in MAE can be challenging.

def mean_absolute_error(true, pred):  
    abs_error = np.abs(true - pred)  
    sum_abs_error = np.sum(abs_error)  
    mae_loss = sum_abs_error / true.size  
    return mae_loss

Mean Bias Error (MBE)

Mean bias error is the tendency of a measurement process to overestimate or underestimate parameter values. The deviation has only one direction, which can be positive or negative. Positive bias means that the error in the data is overestimated, and negative bias means that the error is underestimated. The mean bias error is the average of the differences between the predicted and actual values. This evaluation metric quantifies the overall bias and captures the average bias in forecasts. It is almost similar to MAE, the only difference is that the absolute value is not taken here. This evaluation metric should be handled with care because positive and negative errors can cancel each other out.

advantage:

To check the direction of a model (i.e. whether there is a positive or negative bias) and correct for model bias, MBE is a good metric to use.

shortcoming:

In terms of magnitude, this is not a good measure because the errors tend to compensate for each other.
It is not very reliable because sometimes high individual errors produce low MBE.
As an evaluation metric, it may always be wrong in one direction.

def mean_bias_error(true, pred):  
    bias_error = true - pred  
    mbe_loss = np.mean(np.sum(diff) / true.size)  
    return mbe_loss

Relative Absolute Error (RAE)

The relative absolute error is calculated by dividing the total absolute error by the absolute difference between the mean and the actual value. RAE and expressed as a ratio. The value of RAE ranges from 0 to 1. A good model will have values close to zero, with zero being the best value.

advantage:

RAE can be used to compare models that measure errors in different units.
RAE is reliable because it protects against outliers.

def relative_absolute_error(true, pred):  
    true_mean = np.mean(true)  
    squared_error_num = np.sum(np.abs(true - pred))  
    squared_error_den = np.sum(np.abs(true - true_mean))  
    rae_loss = squared_error_num / squared_error_den  
    return rae_loss

Mean Absolute Percentage Error (MAPE)

The mean absolute percent error is calculated by dividing the difference between the actual and predicted values by the actual value. MAPE, also known as Mean Absolute Percentage Deviation, increases linearly with error. The smaller the MAPE, the better the model performance.

advantage:

MAPE is independent of the size of the variable because its error estimates are in percentages.
All errors are normalized on a common scale and are easy to understand.
MAPE avoids the problem of positive and negative values canceling each other out.

shortcoming:

When the denominator value is zero, we face a "division by zero" problem.
MAPE penalizes errors with smaller values more than errors with larger values.
Because of the division operation, for the same error, the change in the actual value will result in a difference in the loss.

def mean_absolute_percentage_error(true, pred):  
    abs_error = (np.abs(true - pred)) / true  
    sum_abs_error = np.sum(abs_error)  
    mape_loss = (sum_abs_error / true.size) * 100  
    return mape_loss

Mean Squared Error (MSE)

Mean Squared Error Also known as L2 loss, MSE calculates the error by squaring the difference between the predicted and actual values and averaging it over the entire data set. MSE is also called quadratic loss because the penalty is not proportional to the error, but rather to the square of the error. Squared error gives higher weight to outliers, resulting in a smooth gradient for small errors.

MSE can never be negative because the error is squared. Error values range from zero to infinity. MSE grows exponentially with increasing error. A good model has an MSE value close to zero.

advantage:

MSE will result in a gradient descent with only one global minimum.
For small errors, it can effectively converge to the minimum value. There are no local minima.
MSE penalizes models with huge errors by squaring the model.

shortcoming:

Sensitivity to outliers amplifies high errors by squaring them.
MSE is affected by outliers and looks for models that perform well enough at the overall level.

def mean_squared_error(true, pred):  
    squared_error = np.square(true - pred)   
    sum_squared_error = np.sum(squared_error)  
    mse_loss = sum_squared_error / true.size  
    return mse_loss

Root Mean Squared Error (RMSE)

RMSE is calculated by taking the square root of MSE. RMSE is also known as root mean square deviation. It measures the average magnitude of error and focuses on the deviation from the actual value. An RMSE value of zero indicates that the model has a perfect fit. The lower the RMSE, the better the model and its predictions.

advantage:

Easy to understand and easy to calculate

shortcoming:

It is recommended to remove outliers to make it work properly.
will be affected by the size of the data sample.

def root_mean_squared_error(true, pred):  
    squared_error = np.square(true - pred)   
    sum_squared_error = np.sum(squared_error)  
    rmse_loss = np.sqrt(sum_squared_error / true.size)  
    return rmse_loss

Relative Squared Error (RSE)

Relative squared error requires taking the mean squared error and dividing it by the square of the difference between the actual data and the mean of the data.

advantage

Insensitive to the mean and scale of the forecast.

def relative_squared_error(true, pred):  
    true_mean = np.mean(true)  
    squared_error_num = np.sum(np.square(true - pred))  
    squared_error_den = np.sum(np.square(true - true_mean))  
    rse_loss = squared_error_num / squared_error_den  
    return rse_loss

Normalized Root Mean Squared Error (NRMSE)

Normalized RMSE is usually calculated by dividing by a scalar value, it can be done in different ways. Sometimes choosing the interquartile range may be the best choice because other methods are prone to outliers. NRMSE is a good measure when you want to compare models with different dependent variables or modify the dependent variable. It overcomes scale dependence and simplifies comparisons between models at different scales and even between data sets.

RMSE / maximum value in the series  
RMSE / mean  
RMSE / difference between the maximum and the minimum values (if mean is zero)  
RMSE / standard deviation  
RMSE / interquartile range

def normalized_root_mean_squared_error(true, pred):  
    squared_error = np.square((true - pred))  
    sum_squared_error = np.sum(squared_error)  
    rmse = np.sqrt(sum_squared_error / true.size)  
    nrmse_loss = rmse/np.std(pred)  
    return nrmse_loss

Relative Root Mean Squared Error (RRMSE)

RRMSE is the dimensionless form of RMSE, which is the root mean square error normalized by the root mean square value, where each residual is scaled according to its actual value.

Excellent when RRMSE < 10%  
Good when RRMSE is between 10% and 20%  
Fair when RRMSE is between 20% and 30%  
Poor when RRMSE > 30%

def relative_root_mean_squared_error(true, pred):  
    num = np.sum(np.square(true - pred))  
    den = np.sum(np.square(pred))  
    squared_error = num/den  
    rrmse_loss = np.sqrt(squared_error)  
    return rrmse_loss

Root Mean Squared Logarithmic Error (RMSLE)

The root mean square log error is calculated by applying log to the actual and predicted values and then taking their difference. RMSLE is robust to outliers where small and large errors are treated uniformly. Penalize the model more if the predicted value is smaller than the actual value, and penalize the model less if the predicted value is larger than the actual value.

advantage:

Doesn't depend on scale and works on a wide range of scales.
It is not affected by large outliers.
It only considers the relative error between actual and predicted values.

def root_mean_squared_log_error(true, pred):  
    square_error = np.square((np.log(true + 1) - np.log(pred + 1)))  
    mean_square_log_error = np.mean(square_error)  
    rmsle_loss = np.sqrt(mean_square_log_error)  
    return rmsle_loss

Huber Loss

Huber loss is a combination of linear and quadratic scoring methods. It has a hyperparameter delta that can be tuned based on the data. For values above delta, the loss will be linear (L1 loss), and for values below delta, the loss will be quadratic (L2 loss). It balances and combines the good properties of MAE (Mean Absolute Error) and MSE (Mean Square Error).

advantage:

It is differentiable at zero.
Outliers are handled correctly due to linearity above delta.
The hyperparameter delta can be tuned to maximize model accuracy.

shortcoming:

To maximize model accuracy, delta needs to be optimized, which is an iterative process.
It can only be differentiated once.

def huber_loss(true, pred, delta):  
    huber_mse = 0.5 * np.square(true - pred)  
    huber_mae = delta * (np.abs(true - pred) - 0.5 * (np.square(delta)))  
    return np.where(np.abs(true - pred) <= delta, huber_mse, huber_mae)

Log Cosh Loss

Log cosh computes the logarithm of the hyperbolic cosine of the error. This function is smoother than quadratic loss. It works like MSE but is not affected by large prediction errors.

advantage:

Can be differentiated twice everywhere at the same time
Requires less computation than Huber

shortcoming:

Less adaptable because it follows a fixed ratio.
Compared with Huber loss, the derivation is more complex and requires in-depth study.

def huber_loss(true, pred, delta):  
    diff = np.cosh(pred - delta)  
    diff = np.log(diff)  
    return diff.mean()

Quantile Loss

Quantile regression loss function is used to predict quantiles. A quantile is a value that determines how many values in a group are below or above a certain limit. It estimates the conditional median or quantile of the response variable (dependent variable) across the values of the predictor variable (independent variable).