Forecasting Knowledge | Forecasting Technology Process and Model Evaluation
Table of contents
technical process
1) Model training stage: predictive factors and outcomes, plus predictive model for model fitting;
2) Predictive stage: substitute predictive factors into the fitted model to obtain predicted probabilities;
3) Calibration and verification stage: do calibration first , and then verify. Calibration is the readjustment of the model between the predicted probability and the true probability, in order to make the predicted probability as accurate as possible. Finally, the calibration probability was compared with the actual hospitalization probability, and a confusion matrix was constructed to evaluate the prediction effect of the model.
The probability predicted by the model may be very different from the real probability. If left unattended, the application of the model will cause many problems. If you want to deal with it, probability calibration is the most common method, that is, to "pull" the biased probability back to the normal value or "close to" the normal value. The author pointed out that for tree models (such as decision trees, random forests, etc.), the probability generally has more deviations, that is, the calibration is very poor. At this time, most of the models need to be recalibrated.
model evaluation
In machine learning, metrics are used to measure the performance and accuracy of a model. Choosing an appropriate evaluation metric depends on the specific task and data type. The following are some common machine learning evaluation indicators:
Accuracy: Accuracy is one of the most commonly used evaluation indicators, which represents the ratio of the number of samples correctly predicted by the model to the total number of samples. However, accuracy may fail when dealing with imbalanced datasets because it does not take into account the distribution of samples across classes.
Precision and Recall: Precision and recall are usually used together to evaluate a binary classification model. The precision rate indicates the proportion of the samples that the model predicts as positive examples that are actually positive examples, and the recall rate indicates the proportion of the samples that the model correctly predicts as positive examples to all the samples that are actually positive examples. In some cases, we need to make a trade-off between precision and recall, and we can use F1 score as a comprehensive indicator, which is the harmonic mean of precision and recall.
ROC curve and AUC (Receiver Operating Characteristic Curve and Area Under the Curve): The ROC curve evaluates the performance of the binary classification model by plotting the relationship between the true positive rate (TPR) and the false positive rate (FPR) of the model at each classification threshold. performance. AUC represents the area under the ROC curve, ranging from 0 to 1, the closer the AUC is to 1, the better the model performance.
Mean Squared Error (Mean Squared Error, MSE): Mean Squared Error is often used in the evaluation of regression problems to calculate the average squared difference between the predicted value and the actual value. A smaller MSE value indicates that the model predicts more accurately.
Log loss (Log Loss): Log loss is used to evaluate the performance of a probability model, which measures the difference between the probability distribution predicted by the model and the actual label. The smaller the log loss, the closer the model's prediction is to the actual situation.
For unbalanced prediction, the AUC difference between the models is very small and has no distinguishing value, while the area under the PR curve (AP stands for average precision, which is a way to calculate the area under the PR curve) is more helpful for distinction.
References
[1] Reference: Development and validation of predictive models for unplanned hospitalization in the Basque Country: analyzing the variability of non-deterministic algorithms