1. What are MSE, RMSE and MAE
MSE refers to Mean Squared Error (Mean Squared Error), which is a common regression model evaluation index . In a regression model, our goal is to predict a continuous output value, and MSE is used to measure the average of the squared error between the predicted value of the model and the true value.
This formula calculates the square of the difference between the predicted value and the true value for each sample, and then sums and averages the results to obtain a mean squared error value for the evaluated model.
Usually, we hope that the smaller the MSE, the better, that is, the stronger the predictive ability of the model. As for the fact that MSE is not intuitive enough , sometimes we also use its square root to represent the error, that is, the root mean square error (RMSE).
When evaluating a model, it is common to randomly split the dataset into training and testing sets. The training set is used to train the model, and the test set is used to evaluate the predictive performance of the model on unknown data. Therefore, the evaluation of the model is meant to be performed on the test set .
MAE refers to the mean absolute error (Mean Absolute Error), which is a common regression model evaluation index. Similar to MSE, MAE is used to measure the gap between the predicted value of the model and the real value. The difference is that MAE calculates the average value of the absolute value of the difference between the predicted value and the true value of each sample.
The smaller the MAE , the better the predictive ability of the model.
In the case of the same MAE, RMSE can show its slight difference.
Comparison of MAS and MSE:
Both MAS and MSE are commonly used indicators for evaluating regression models. If there are outliers (i.e. outliers ) in the data set, then MAE is more suitable than MSE as an indicator for evaluating regression models. It is more robust because it can better Handle outliers. However, if there are no outliers or few outliers in the data set , MSE may be more sensitive and better reflect the nuances of the model , so it is more appropriate to use MSE as an evaluation indicator in this case.
What is robustness:
Robustness refers to the ability of a system or method to change input changes such as noise, outliers , missing data , and parameter disturbances. In fields such as machine learning, data mining, and statistics, robustness usually refers to the stability and reliability of an algorithm to changes in input data .
For example, in regression problems, a model is less robust if it is very sensitive to changes in the input of a certain value. A model is more robust if it can handle noise or outliers well. Similarly, in clustering and classification problems, a model can run stably under the absence of data or input parameter disturbance, and maintain high performance under different input and parameter changes, which also shows that it has strong robustness .
In practical applications , robustness is one of the very important properties. Because in actual scenarios, the data is usually imperfect , and there are problems such as outliers and noise . A robust model or algorithm can reduce the impact of data problems and improve the reliability and generalization performance of the model.
2. Code example
The following is a code example that uses three gradient descent methods of BGD, SGD and MBGD to perform unary linear regression, and compares the three methods based on the MSE and RMSE indicators.
#train_test_split 函数将数据集划分为训练集和测试集
from sklearn.metrics import accuracy_score
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
from sklearn import metrics
# Load iris dataset and extract two features
iris = load_iris()
X = iris.data[:, :2] # Only consider two features
y = iris.target
# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Batch Gradient Descent
model_bgd = SGDRegressor(learning_rate='constant', eta0=0.01, max_iter=1000, tol=1e-3)
model_bgd.fit(X_train, y_train)
# Stochastic Gradient Descent
model_sgd = SGDRegressor(learning_rate='constant', eta0=0.01, max_iter=1000, tol=1e-3)
model_sgd.fit(X_train, y_train)
# Mini-batch Gradient Descent
model_mbgd = SGDRegressor(learning_rate='constant', eta0=0.01, max_iter=1000, tol=1e-3)
model_mbgd.fit(X_train, y_train)
# Make predictions on testing set
y_pred_bgd = model_bgd.predict(X_test)
y_pred_sgd = model_sgd.predict(X_test)
y_pred_mbgd = model_mbgd.predict(X_test)
# Compute mean squared error for each model MSE
mse_bgd = mean_squared_error(y_test, y_pred_bgd)
mse_sgd = mean_squared_error(y_test, y_pred_sgd)
mse_mbgd = mean_squared_error(y_test, y_pred_mbgd)
#Compute RMSE 均方根 误差
rmse_bgd = np.sqrt(mse_bgd)
rmse_sgd = np.sqrt(mse_sgd)
rmse_mbgd = np.sqrt(mse_mbgd)
# Print mean squared error for each model
print("Batch Gradient Descent MSE : ", mse_bgd)
print("Stochastic Gradient Descent MSE : ", mse_sgd)
print("Mini-batch Gradient Descent MSE: ", mse_mbgd)
# Print mean Root squared error for each model
print("Batch Gradient Descent RMSE : ", rmse_bgd)
print("Stochastic Gradient Descent RMSE : ", rmse_sgd)
print("Mini-batch Gradient Descent RMSE: ", rmse_mbgd)
# Plot training data and regression lines
plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=plt.cm.Set1, edgecolor='k')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
# Batch Gradient Descent
m_bgd = model_bgd.coef_[0]
b_bgd = model_bgd.intercept_
plt.plot(X_train[:, 0], m_bgd * X_train[:, 0] + b_bgd, color='blue', linewidth=2)
# Stochastic Gradient Descent
m_sgd = model_sgd.coef_[0]
b_sgd = model_sgd.intercept_
plt.plot(X_train[:, 0], m_sgd * X_train[:, 0] + b_sgd, color='red', linewidth=2)
# Mini-batch Gradient Descent
m_mbgd = model_mbgd.coef_[0]
b_mbgd = model_mbgd.intercept_
plt.plot(X_train[:, 0], m_mbgd * X_train[:, 0] + b_mbgd, color='green', linewidth=2)
plt.legend(['Batch GD', 'SGD', 'Mini-batch GD'])
plt.show()
# accuracy1 = r2_score(y_test, y_pred_bgd)
# print("Batch Gradient Descent精确率：", y_pred_mbgd)
accuracy1 = metrics.r2_score(y_test, y_pred_bgd)
print("Batch Gradient Descent精确率：", accuracy1)
accuracy2 = metrics.r2_score(y_test, y_pred_sgd)
print("Stochastic Gradient Descent精确率：", accuracy2)
accuracy3 = metrics.r2_score(y_test, y_pred_mbgd)
print("Mini-batch Gradient Descent精确率：", accuracy3)operation result:
Summarize
After the above code is executed, the performance evaluation indicators of BGD, SGD and MBGD calculated by using MSE and RMSE on the training set and test set will be output. After multiple runs and comparisons: for the iris data set, we can see that the MBGD method is more accurate. It is more stable , compared with the BGD and SGD methods, it has achieved lower MSE and RMSE values on the test set , so we can think that the performance of the MBGD method is better.
references
[EB/OL].https://www.bilibili.com/video/BV1wi4y157Tt/?spm_id_from=333.337.search-card.all.click&vd_source=ce838d56b689e47d2bdd968af2d91d20,2020-12-2
[EB/OL].https://www.bilibili.com/video/BV1Qi4y157BF/?spm_id_from=333.788.recommend_more_video.0&vd_source=ce838d56b689e47d2bdd968af2d91d20,2020-12-20