Evaluation of Machine Learning Regression Task Indicators and Sklearn Neural Network Model Evaluation Practice

Machine learning regression model evaluation refers to the evaluation of the performance of the regression model in order to select the best regression model. Among them, MAE, MSE, and RMSE are used to measure the error between the model's predicted value and the real value, and R² is used to measure the degree of fit of the model to the data. In practical applications, we can use these indicators to evaluate the performance of the regression model and optimize the model.

For example, in the industrial field, regression algorithms can predict production indicators such as electricity load and power generation through regression analysis of historical data.

1. Evaluation of prediction results of machine learning regression models

There are many methods for evaluating the prediction results of machine learning regression models. The following are some commonly used methods:

R2 (coefficient of determination): R2 evaluates the fit of the model to the data. Its value ranges from 0 to 1. The closer to 1, the better the fit of the model (the score used for sklearn regression model evaluation defaults to R2. calculation formula). The calculation formula of R2 is as follows:

$$R2\_=1-\frac{{\sum }_{i=1}^n(y_i-\widehat{y_i}{)}^2}{{\sum }_{i=1}^n(y_i-\overline{y_i}{)}^2}$$

The best score for the R2 (coefficient of determination) regression scoring function is 1.0, which can be negative (since the model can vary arbitrarily).

Mean Absolute Error (Mean Absolute Error, MAE): MAE is the average of the absolute value of the difference between the predicted value and the true value, and the smaller its value, the higher the accuracy of the model prediction. The calculation formula is:

$$MAE=\frac{1}{n}\sum _{i=1}^n\mid y_i-\widehat{y_i}|$$

Also known as L1 norm loss, to measure the real distance between the predicted value and the real value.

Mean Squared Error (Mean Squared Error, MSE): MSE is the average of the square of the difference between the predicted value and the true value, and the smaller its value, the higher the accuracy of the model prediction. The calculation formula is:

$$MSE=\frac{1}{n}\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2$$

The square form is convenient for derivation, so it is often used as the loss function of linear regression.

Root Mean Squared Error (Root Mean Squared Error, RMSE): RMSE is the square root of MSE, and the smaller its value, the higher the accuracy of the model prediction. The calculation formula is:

$$RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2}$$

It is often used as a standard for measuring the prediction results of machine learning models, and measures the deviation between the observed value and the true value. It is greatly affected by outliers, and its robustness is relatively poor.

Among them, $y_i$​Is the real value, $\widehat{y_i}$​Is the predicted value, $\overline{y_i}$​is the mean of the true values, and n represents the sample size.

R-squared (R-squared): R-squared is calculated in a similar way to R2, but it can punish the complexity of the model to avoid model overfitting.

Adjustable coefficient of determination (Adjusted R-squared): The coefficient of determination is an improved version of R-squared, which takes into account the influence of the number of independent variables on the model, and can more accurately evaluate the degree of fitting of the model.

2. How to evaluate machine learning regression models

The regression model obtained by machine learning can be evaluated from many aspects, such as: the degree of fitting of the model, the predictive ability of the model, the stability of the model, etc.

Among them, the fitting degree of the model can be measured by the coefficient of determination (Coefficient of Determination, R2). The value range of R2 is between 0 and 1, and the closer to 1, the better the model fits the data. However, R2 does not tell us which variables have an effect on the target variable.

The predictive ability of the model can be measured by indicators such as root mean square error (Root Mean Squared Error, RMSE), mean absolute error (Mean Absolute Error, MAE). These metrics can be used to measure the predictive power of the model on new data.

The stability of the model can be measured by methods such as cross validation (Cross Validation). Cross-validation can be used to evaluate the performance of the model on different data sets to determine whether the model is overfitting or underfitting.

3. Sklearn regression model practice

3.1. Multilayer perceptron regression model

The practical example uses a simple neural network perceptron (MLPRegressor).

from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error
from sklearn.utils import column_or_1d
import pandas as pd
import numpy as np
import joblib
import matplotlib.pyplot as plt

df = pd.read_csv('power_pv09.csv')
# 数据处理过程，略
train_datas = df[cols_dict].loc[df['p_mp'] > 0]
y = train_datas[['ThisPower']] 
X = train_datas.drop(columns='ThisPower')
model_filename = 'mlpmodel.joblib'

x_train, x_test, Y_train, Y_test = train_test_split(X, y, test_size=0.2)

std_x = StandardScaler()
std_x.fit(x_train)
# y值可以不进行归一化，使用原值
std_y = StandardScaler()
std_y.fit(Y_train)

y_train = std_y.transform(Y_train)
y_test  = std_y.transform(Y_test)
X_train = std_x.transform(x_train)
X_test = std_x.transform(x_test)

forest_model = MLPRegressor(hidden_layer_sizes=(80, 90, 80), 
                            solver = 'adam', 
                            tol=1e-3, max_iter=600, random_state=0).fit(X_train, y_train)

joblib.dump(forest_model, model_filename)  # 存储模型

y_ = forest_model.predict(X_test)
y_ = y_.reshape(-1,1)
pre_y = std_y.inverse_transform(y_)

3.2. Visualize the training process

# 可视化损失函数
plt.figure(figsize=(12,6))
plt.plot(forest_model.loss_curve_)
plt.xlabel("iters")
plt.ylabel(forest_model.loss)
plt.show()

Note: for solver='adam' etc. Newton's method is not applicable for solver='lbfgs'.

3.3. Regression Model Evaluation

Use common machine learning regression model evaluation methods such as R2, MSE, and MAE, and call their functions in the Sklearn library to achieve evaluation. For details, see the following code.

R2 = forest_model.score(X_test, y_test)
print ("Test R2 Score = ", R2)
'''计算训练集 MSE'''
pre_train = forest_model.predict(X_train)
mse_train = mean_squared_error(pre_train,y_train)
print ("Train mean squared ERROR = ", mse_train)
'''计算训练集 MAE'''
pre_train = forest_model.predict(X_train)
mae_train = mean_absolute_error(pre_train,y_train)
print ("Train mean absolute ERROR = ", mae_train)
'''计算测试集mse'''
pred_test = forest_model.predict(X_test)
mse_test = mean_squared_error(pred_test,y_test)
print ("Test mean_squared ERROR = ", mse_test)
'''结果可视化'''
xx=range(0,len(y_test))
plt.figure(figsize=(8,6))
plt.scatter(xx,Y_test,color="red",label="Sample Point",linewidth=3) 
plt.plot(xx,column_or_1d(pre_y, warn=True),color="blue",label="Fitting Line",linewidth=2)
plt.legend()

The evaluation results are as follows:

Test R2 Score = 0.884
Train mean squared ERROR = 0.0475
Train mean absolute ERROR = 0.147
Test mean_squared ERROR = = 0.1026

The comparison chart between the test set prediction results and the real value is as follows:

reference:

Big Fish. Common Evaluation Indicators for Machine Learning . Zhihu. 2022.11
Xiaofei@IDO. Machine Learning: Evaluation Indicators for Regression Models . CSDN Blog. 2021.06